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BACKGROUND INFORMATION ABOUT THE SERVE CENTER 


The SERVE Center at the University of North Carolina at Greensboro (UNCG) is a university- 
based research, development, dissemination, evaluation, and technical assistance center. Its 
mission is to support and promote teaching and learning excellence in the K-12 education 
community. 


Since its inception in 1990, SERVE has been awarded over $200 million in contracts and grants. 
It has successfully managed 14 major awards including four consecutive contracts for the 
Regional Educational Laboratory for the Southeast (REL-SE) funded by the Institute of Education 
Sciences (IES) at the US Department of Education (USED) and four awards from USED for the 
National Center for Homeless Education (NCHE). In addition, past SERVE awards include a five- 
year Technology Grant for Coordinating Teaching and Learning in Migrant Communities, three 
consecutive contracts as the Eisenhower Consortium for Mathematics and Science Education 
for the Southeast, and two consecutive Regional Technology in Education Consortium grants. 


At the national level, SERVE operates the National Center for Homeless Education (NCHE), 
USED’s technical assistance and information dissemination center in the area of homeless 
education. NCHE uses state-of-the-art technology for web communication and online 
professional development and for supporting state coordinators of homeless education, local 
program coordinators, educators, parents, and advocates in all 50 states and in 15,000 school 
districts. 


In addition to national-level NCHE activities, SERVE currently conducts research studies and 
evaluations under grants and contracts with federal, state, and local education agencies. 
Examples of SERVE’s grant-funded research work include three federally funded studies of the 
impact of Early College High Schools and a five-year study of the impact of dual enrollment in 
Carolina. Contract work includes evaluations of four Investing in Innovation (i3) projects, the 
Winston-Salem/Forsyth County Magnet Program in North Carolina, the Guilford County Schools 
teacher incentive program (Mission Possible), the USED-funded Bridges to Early Learning 
Project in South Carolina, and North Carolina’s Race to the Top Initiative. The Guiding Principles 
for Evaluators (American Evaluation Association, 2004) and the What Works Clearinghouse 
Standards (Institute of Education Sciences, October, 2017) guide the evaluation work 
performed at the SERVE Center. 
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IMPROVING HIGH SCHOOL THROUGH STEM INSTRUCTION AND EARLY COLLEGE STRATEGIES: 
THE IMPACT OF THE STEM EARLY COLLEGE EXPANSION PARTNERSHIP (SECEP) 


SECTION I: INTRODUCTION AND OVERVIEW 


The changing U.S. economy means that jobs that pay a living wage are more likely to require 
some form of postsecondary education, particularly in fields related to science, technology, 
engineering, or mathematics, known as STEM for short (Carnevale & Desrochers, 2003; 
Carnevale, Smith, & Strohl, 2010). Yet, concerns remain that too few students are successfully 
earning postsecondary credentials, particularly in STEM areas. The problem is particularly acute 
for certain populations of students, including low-income students, students who are the first in 
their family to go to college, and students who are members of racial and ethnic groups 
underrepresented in college. In response to these concerns, educators and policymakers have 
been exploring a variety of approaches at the high school level to increase students’ interest 
and skills in STEM and to increase their likelihood of enrolling and succeeding in postsecondary 
education. 


One of the most successful efforts to increase students’ enrollment and success in 
postsecondary education has been the Early College. As originally conceptualized, Early Colleges 
are small schools focused purposefully on college readiness for all students. Frequently located 
on college campuses, Early Colleges target students who might face challenges in 
postsecondary education. Early Colleges serve students starting in 9"" grade; the goal is to have 
students graduate in four or five years, with a high school diploma and either a postsecondary 
credential (usually an associate degree) or two years of transferable college credit. Supported 
by an initial investment by the Bill and Melinda Gates Foundation, the small Early College 

Model expanded across the country. 


This model has been the subject of three rigorous longitudinal experimental studies funded by 
the U.S. Department of Education and led by SERVE Center at UNCG and an experimental study 
conducted by the American Institutes of Research. These studies found that the Early College 
Model had positive impacts on a variety of outcomes, including staying in school, progressing in 
college preparatory courses, graduating from high school, and enrolling in and graduating from 
college (Berger et al., 2013; Edmunds, Bernstein, Unlu, Glennie, & Smith, 2013; Edmunds et al., 
2012; Edmunds, Unlu, et al., 2017; Edmunds, Willse, Arshavsky, & Dallas, 2013). 


As implemented, the Early College is not necessarily required to have a focus on STEM, 
although some Early Colleges have structured themselves around STEM themes. There is little 
research, however, on how STEM can be successfully integrated into the Early College Model. 
Additionally, there is little research that indicates whether and how the small Early College 
Model can be scaled to comprehensive high schools. Studies undertaken of two recent efforts 
to implement the model in comprehensive high schools have found that the schools can make 
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some changes, although it is challenging work (Edmunds, Klopfenstein, Lewis, & Hutchins, 2018; 
Edmunds, Naumenko, Henson, & Hutchins, 2017). The STEM Early College Expansion 
Partnership (SECEP) is among the first set of large-scale efforts to apply Early College strategies 
in comprehensive high schools and the first one to attempt to simultaneously integrate STEM 
instructional and curriculum changes. 


Supported by a $12 million grant from U.S. Department of Education’s Investing in Innovation 
(i3) Program, SECEP is designed to 


..improve STEM education for 22,000 high-need middle and high school students, 
decreasing drop-out rates and boosting college enrollment in a number of districts in 
Connecticut and Michigan. SECEP will further improve underrepresented populations’ 
access to STEM careers by increasing the number of students enrolling in dual credit 
STEM courses and pursuing postsecondary credentials (SECEP Year 2 Management 
Plan). 


SECEP took place in two areas: Bridgeport, Connecticut and four intermediate school districts 
(ISDs) in Michigan. The project was a partnership between three primary organizations, each 
with a long history of Early College work. The overall effort was led by the National Center for 
Restructuring Education, Schools and Teaching (NCREST) at Teachers College, Columbia 
University, collaborating with Jobs for the Future (JFF), and the Middle College National 
Consortium (MCNC). JFF was responsible for project implementation in Bridgeport, and MCNC 
was responsible for project implementation in Michigan. 


SECEP set a bold goal for itself: 90% of students in SECEP schools would graduate high school 
with some college credit. This goal was intended to promote broader change in the schools, 
focusing on four Design Principles that are described in more depth in Section II of the report. 


This report presents findings on the impact of this five-year project, including results for schools 
and students. The next section of the report describes the STEM Early College Model in more 
detail. Section III includes the methodology used to assess the impacts on schools and an 
overview of the student impact methodology. Section IV utilizes survey and interview data to 
describe the impact of the project on schools. Because of differences in data availability that 
impacted the study design, student impacts for Michigan and Connecticut are reported in 
separate sections. Section V describes the student impact analysis methodology and results for 
Michigan and Section VI describes the methodology and results for Connecticut. The report 
ends with key conclusions and lessons for educators to consider as they move forward with this 
work. 
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SECTION II: THE STEM EARLY COLLEGE HIGH SCHOOL MODEL AND ITS SUPPORTS 


SECEP’s goal was to redesign high schools by focusing on STEM curriculum and instruction in 
schools while also expanding access to college courses for students. The project team did this 
through supporting the implementation of the STEM Early College High School Model in 
comprehensive high schools. 


STEM Early College High School Model 


The STEM Early College High School Model, as articulated by NCREST and its partners, included 
four Design Principles: (1) a STEM College-Focused Academic Program, (2) Student Support, (3) 
High School-College Collaboration, and (4) a Culture of Continuous Improvement. To support 
implementation, the SECEP project staff developed a rubric that described implementation of 
these Design Principles. These Design Principles are described in this section and are also 
operationalized more fully in the section on school-level impacts. 


STEM College-Focused Academic Program (Design Principle #1) 


According to the SECEP Design Principles rubric, the first Design Principle was defined as: 
“Schools implement comprehensive, standards-based, STEM-focused curricula that provide a 
bridge to STEM postsecondary studies.” 


One of the core expectations of the model was that students have access to college courses 
while in high school, with an emphasis on STEM-oriented pathways. Schools were expected to 
implement a 4-5-year academic plan that included a sequence of STEM and non-STEM courses, 
preparing students for college and providing opportunities for college credit. The intent was 
that these pathways would give students a head start on specific college majors or prepare 
students for careers, primarily focused on STEM. The project encouraged schools to create 
systems that would allow students to earn at least 12 college credits by the time they 
graduated from high school. 


In addition to curricular emphases, there was also an emphasis on changing instruction in two 
areas: college readiness and STEM. Schools were expected to ensure that students were ready 
for college courses by increasing the rigor of instruction and embedding key college preparatory 
and college navigational skills into high school courses. Teachers were also expected to utilize 
STEM instructional practices, such as inquiry-based learning and problem- and project-based 
learning. 


Student Support (Design Principle #2) 


Under this Design Principle, “Supports are offered to all students to meet their academic, social, 
and emotional needs in high school and the transition to college.” This Design Principle 
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recognizes that the rigor of the student experience cannot be increased without accompanying 
supports that will help students be successful. 


The expectation was that each school would develop a system of supports tied to students’ 
needs and their academic progression. As noted in the rubric, “High schools support students: 
(1) to stay on track academically, (2) to prepare for college, (3) in STEM coursework, (4) in 
college coursework, (5) in meeting socio-emotional needs.” These supports also included 
counseling relative to career readiness and to expose students to STEM careers. 


High School-College Collaboration (Design Principle #3) 


The Early College Model also depends on close collaboration between the district and its 
college partner(s), including ongoing communication and shared resources. According to the 
rubric, “High schools and colleges take joint responsibility for students’ educational success.” 
Each school or district was expected to have a formal Memorandum of Understanding with at 
least one college partner that delineated roles and responsibilities for each partner. There were 
also expectations regarding ongoing communication between the partners and continuing 
collaboration around courses and supports. 


Culture of Continuous Improvement (Design Principle #4) 


The fourth Design Principle includes an emphasis on using data to make decisions as well as 
aligning the high school and postsecondary curriculum: “High schools and colleges engage in 
evidence-based discussions to continuously improve students’ experiences and outcomes.” 
The intent was that each school create processes by which data were being used to inform 
instruction. Additionally, schools were expected to offer teachers regular and ongoing 
professional development related to the goals of the project and support ongoing teacher 
collaboration. 


Implementation Supports 


To support implementation of the STEM Early College Model in school, SECEP partners provided 
a suite of services including: (1) leadership coaching to districts, (2) workshops and conferences, 
(3) a Community of Practice, (4) school-based SECEP coaching, (5) district-level teams, and (6) 
district-college collaboration. Each is described in more depth below. 


Leadership Coaching to Districts (Implementation Support #1) 


Staff from the SECEP partners (JFF in Bridgeport and MCNC in Michigan) provided leadership 
coaching and technical assistance to districts to help them develop the capacity to plan, 
monitor, and implement the STEM Early College Model in their district. Part of this technical 
assistance included support around the development of an implementation plan that aligned 
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the multiple initiatives within each district and training for individuals who would serve as 
SECEP coaches directly to the schools. 


Data collected for the evaluation indicate that leadership coaching and technical assistance to 
the districts took various forms: monthly meetings with key district representatives, assistance 
with developing a strategic plan, individual coaching for district staff, collecting data during site 
visits, reviewing data with districts leaders, problem solving, monitoring implementation, 
developing annual plans, and developing workshops. 


All districts were provided with a template to complete annual plans intended to guide project 
implementation. District staff noted in their reports that they worked with project teams on 
these plans. The plans (1) focused on the SECEP Design Principles; (2) examined the 
implementation process, professional development needs, school structures, and resources; (3) 
named persons accountable for implementation; and (4) used evidence to show whether they 
had achieved their goals. 


The project staff also worked with district staff to support collaboration between the districts 
and their college partners; more specifically, the development of academic plans, curricula, 
course sequences, scope and sequences, and STEM pathways. 


The expectation was that each district would receive an average of 22 contacts annually. This 
expectation was met each year and, in many cases, substantially exceeded. For example, on 
average, in the fourth year of implementation, the project staff had over 165 substantive 
technical assistance interactions with the district staff, although there was substantial range 
across districts from a low of 34 interactions in one district to a high of 432 in another. 


Workshops and Conferences (Implementation Support #2) 


Training is seen as an important part of implementation, providing people exposure to 
important information (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). SECEP partners 
provided workshops and conferences that focused on different components of the STEM Early 
College Model. Participants from both states attended the MCNC Summer Institute annually. At 
that conference, the SECEP districts brought a team and attended sessions focused on 
developing Early/Middle College Programs, academic support for students, instructional 
strategies, and analysis of SECEP data. This conference also provided teams time to reflect on 
the previous year and begin planning for the next academic year. Each state also had their own 
regional/state conferences that included topics such as: implementing project-based learning, 
instructional strategies, developing effective lessons, implementing engineering thinking and 
signature pedagogical approaches to STEM instruction, and assessment. 


A key activity during the national and state conferences was modeling the use of data to guide 
program improvement. NCREST staff collected and analyzed formative data to share with the 
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districts and schools on an ongoing basis. These formative data consisted of a student survey, 
publicly available data, outcomes data, and college enrollment data, and the staff survey results 
from the external evaluation. The NCREST team facilitated data discussions with the schools, 
helping them understand how the data could inform their work. 


Finally, there were professional development opportunities at the local level that were 
scheduled according to needs identified by the schools and districts. These opportunities were 
provided by a range of individuals and groups, including local teachers or administrators, 
district staff, Teachers College professors/staff, or external organizations that provided relevant 
training. The topics primarily focused on STEM instruction and included science and math 
integration, integrating technology, instructional rounds, and inquiry science training. 


Participants in the professional development included both district and school staff, who were, 
respectively, expected to participate in at least five days of professional development annually. 
This expected level was met or exceeded each year. For example, in the fourth year of 
implementation, district teams participated in an average of 27 days of professional 
development with participation ranging from a low of 10 days in one district to a high of 80 in 
another. School teams participated in an average of 14 days of professional development with a 
low of 6 and a high of 39 days. 


Community of Practice (Implementation Support #3) 


One of the project’s supports was the creation of a common, online Community of Practice 
(COP) where resources and ideas could be shared across states and districts. The website that 
hosts this community was launched in December 2015 by JFF; it provided links to resources, 
experts, webinars, and calendars relevant to the SECEP project. The site represented the online 
community of districts involved in SECEP as well as other districts doing Early College reform 
under JFF guidance. As part of the COP, JFF staff and their guest presenters delivered a series of 
webinars focused on topics such as project-based learning, technology integration, growing the 
number of dual enrollment teachers, data use, redesigning the high school experience, school 
culture, work-based learning, and instructional coaching. The COP has archived these webinars 
and other resources on topics such as coaching, instructional practices, and sustainability. 


School-Based SECEP Coaching (Implementation Support #4) 


Another key support was the provision of SECEP coaching services directly to the schools. 
Coaching that follows up on professional development is important to the success of program 
implementation (Fixsen, Blase, Naoom, & Wallace, 2009; Joyce & Showers, 2002). Intended to 
provide support around implementation of the SECEP Model, SECEP coaching was structured 
very differently in the two states. 
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In Bridgeport, the model included a JFF instructional coach, a JFF leadership coach, coaches 
from Teachers College, and district- and school-based coaches. The JFF leadership coach 
worked with principals and their leadership teams in schools, reviewing data and progress 
toward the SECEP deliverables. The JFF instructional coach worked with district staff, including 
district-based instructional coaches, and provided professional development and support. This 
instructional coach worked with school staff on implementation of the Common Instructional 
Framework, a set of six student-centered instructional practices designed to prepare students 
for college. During the fourth year of the project, this coach also began working with school 
leaders. The coaches from Teachers College worked with school staff around implemented 
STEM-focused instructional practices. 


Bridgeport also had school-based teacher leaders and instructional coach consultants. The 
teacher leaders only worked in their home school. In Years 2 and 3, two district-level 
consultants each worked with three schools, providing support to six out of nine project 
schools. However, by the middle of Year 4, there was only one district consultant working in 
three project schools. 


In Michigan, the coaching was carried out by staff from the Intermediate School District (ISD) or 
the local district, who used a mix of training, technical assistance, and instructional coaching to 
support school staff. These staff members coached teachers at the school, facilitated SECEP 
school team meetings, and brought additional resources. In addition to district support, all the 
ISDs either employed someone to provide school-level instructional coaching full time or had 
their teacher leaders providing coaching or assistance to other teachers. Across the ISDs, the 
coaches were former or current classroom teachers and/or administrators familiar to the staff. 
When the schools did not have an internal coach, the ISD provided coaches that worked with 
the professional learning teams and individual teachers as needed and facilitated leadership 
meetings. 


The expectation was that each school was to receive at least 12 coaching days annually in Years 
2 and 3, with a reduction in Year 4 to 10 days per high school and 2 days per middle school. This 
expectation was not met for Year 2, but it was met for Years 3 and 4. In the fourth year of 
implementation, the high schools received an average of 19 days of coaching support with the 
amount ranging from 6 to 34 days. The middle schools received an average of 10 days of 
coaching, ranging from a low of 2 days to a high of 34 days. 


District-Level SECEP Implementation Teams (Implementation Support #5) 


Having a core group of individuals who meet together regularly to support implementation is 
an important component of implementation (Meyers, Durlak, & Wandersman, 2012). Each ISD 
or district was expected to create a SECEP team that would serve as an organizing structure for 
implementation. Each district created teams which met regularly and discussed grant-related 
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issues such as information on relevant professional development, and general district or state 
policy that might impact the SECEP work. In interviews, respondents noted that the SECEP team 
was an important implementation tool, which (1) provided networking and support to school 
and district leaders, (2) helped to share information and served as a resource, (3) created a 
sense of urgency and priority for the project, and (4) provided accountability for the school 
leaders. 


District-College Collaboration (Implementation Support #6) 


The final key support of the grant was district-college collaboration, whereby ISDs and/or 
districts developed postsecondary partnerships and pathways. Each local district had a formal 
Memorandum of Understanding (MOU) with at least one college partner. Each district also 
developed a series of pathways that aligned high school and college courses. For example, one 
district had pathways that led to degrees and certificates in: Computer Occupations 
Technology—Applications Developer, Computer Occupations Technology—Computer Security, 
Nursing, Automotive Technology, and Industrial & Manufacturing Technology. 


SECEP Logic Model 


The implementation supports were intended to promote implementation of the Design 
Principles in schools, which were then expected to lead to improved student outcomes. The 
SECEP Logic Model (Figure 1) is a graphic representation of the project’s activities, the changes 
expected to occur at the school level, and the longer-term student outcomes the program was 
designed to impact. This logic model guided the work of the evaluation. 
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Figure 1. SECEP Logic Model 
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School-Level Implementation of 


STEM Early College Design Principles 


STEM College-Focused Academic 

Program 

¢STEM-focused curricula 

¢ Instructional rubrics that support 
college readiness 

¢lmproved pedagogical practices 

¢ Aligned STEM pathways 


¢4-5 year academic program that 
includes college coursework 


Student Support 
Comprehensive academic and social 
programming and supports 


¢ Support for students’ development of 
college knowledge 


¢Schools and college collaborative 
efforts to support students 


High School-College Collaboration 
¢ Shared resources 
© Formal and informal communication 


Culture of Continuous Improvement 
© Collection and shared use of district 
and college data 


¢ Work on aligning curriculum to 
college expectations 


Student Outcomes 


10 percentage point 
increase in students 
taking and succeeding in 
college preparatory 
courses and STEM 
Courses 


90% of students have 
received some college 
credit. 


Cumulative dropout 
rates are 5 percentage 
points lower. 


Long term: 10 
percentage point higher 
rate of graduation. 


Long term: Increase in 
students pursuing 
postsecondary/ college 
credentials/ degrees in 
STEM 


SECTION III: EVALUATION METHODOLOGY 
The impact study was designed to answer two key research questions: 


1. What is the impact of SECEP on schools’ implementation of the STEM Early College 
Design Principles? 

2. What is the impact of SECEP on key student outcomes related to college-level 
coursetaking and remaining in school? 


This section describes the procedures used to assess the impact on schools’ implementation of 
the Design Principles and provides an overview of the approach used to assess student 
outcomes. The state-specific student impact methodologies are described in more depth in 
their respective sections along with the results. 


Examining Impact on Schools’ Implementation of the STEM Early College Design Principles 


As described in Section II, schools were expected to implement the four SECEP Design 
Principles, which, in turn, were expected to lead to improved student outcomes, a primary goal 
of this grant. The implementation of the Design Principles was evaluated via a staff survey and 
interviews with district, college, and school staff during site visits. The methodology for the 
survey and site visits is described in the next two sections. 


Staff Survey Methodology 


Over the course of the grant, the SECEP staff implementation survey was administered four 
times. The first administration occurred between late January and early May of 2015. The 
second and third administrations occurred between early February and late March of 2016 and 
2017, respectively, while the fourth occurred during February of 2018. The online survey was 
administered to all staff at program schools and to only the principals (or designated staff) at 
matched comparison schools. 


Staff Survey Content 


The survey was primarily intended to measure staff perceptions of the implementation of the 
SECEP Design Principles in each school. Additionally, the survey measured program 
implementation components such as: (1) buy-in and understanding of the program; (2) 
participation in, and perceptions of, professional development and coaching; and (3) perceived 
impacts of the program. Given that the Design Principle components (e.g., college readiness 
and project-based learning) and implementation components (e.g., professional development 
and coaching) could be implemented in schools without being directly involved in SECEP, most 
questions (such as those administered to both treatment and comparison schools) were 
structured without mention of the SECEP project. 
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Most survey items were analyzed as scales, with a few items analyzed individually. Table III- 1 


provides an overview of the survey scales, including sample items on each scale, response 


options, and reliability. To determine reliability, Cronbach's a@ was calculated using responses 


from the first two survey administrations (reliability analysis suggests moderate to good 


reliability for most scales with Cronbach’s alpha values ranging from .76 to .92). When teachers 


and administrators answered the same questions, data from all staff were used; in all other 


cases, only the teachers’ data were used to determine the scale reliability. A table listing all 


items on each scale can be found in Appendix A. 


Table III-1. Summary of Survey Scales’ Properties and Questions 


1“T” indicates teacher items and “A” indicates administrator items. 


SERVE 


Staff Scale a Sample Questions Response Options 
SECEP Program .88 e | understand the goals and objectives of the SECEP program that is 1 = Strongly Disagree 
Understanding and being implemented in our school. 2 = Disagree 

Buy-in e  Allstaff in this school share a goal of preparing every student for 3 = Agree 
postsecondary education without remediation. 4 = Strongly Agree 

College-Going 81 e Most faculty and staff in this school expect every student to receive 

Expectations at least some postsecondary education or training. 

e Our school explicitly and purposefully focuses on building students’ 
postsecondary aspirations. 

College Readiness 84 e The school has clearly articulated goals for academic college 1 = Not True at All 
readiness (such as content knowledge, writing, reading, and critical 2 = Somewhat True 
thinking skills, etc.). 3 = Mostly True 

e Staff incorporates measures of academic college readiness into their | 4 = Entirely True 
classroom. 

School Academic 85 e Our school has a system to identify students in need of academic 

and Social- support. 

Emotional Support e Our school has a system to provide support to students taking 
college courses. 

Staff Academic and 78 e |regularly provide academic supports to students who need it. 1 = Not True at All 

Social-Emotional ¢ Most staff regularly provide academic supports to students who 2 = Somewhat True 

Support (T, A) need it. 3 = Mostly True 

4 = Entirely True 
College Readiness .76 (T)? In the current school year, how often have you: 1=Never 
Skills .92 (A) e Worked with students on time management and study skills? 2=A Few Times This Year 
e Encouraged students to advocate for themselves with high school 3=Once or Twice a Month 
and college faculty? 4=Once or Twice a Week 

Rigorous 84 In the current school year, how often have you: 5=Almost Every Day 

Instruction e Asked students to defend their own ideas or point of view in writing 
or ina discussion? 

e Asked students to analyze or interpret documents or data? 

Inquiry, Project- 82 In the current school year, how often have you: 

Based Learning e Had students develop their own questions and then answer them? 

e Implemented projects in your classrooms? 

High School and .90 This school year, how often did you meet with college faculty: 1 = Never 

College e to understand college expectations in my subject area. 2=1-2Times 

Collaborations e to improve student opportunities to study in STEM areas. 3 =3-6 Times 

4 = More Than 6 Times 

Local College 89 To what extent have you or your school received the following 1=None 

Support resources/supports from your local college over the past year? 2 =Alittle 

e Research or internship opportunities 3 =A fair amount 
e College courses 4=A lot 
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Staff Survey Sample 


The online survey was administered to certified staff in treatment schools and to administrators 
(or their surrogate) in comparison schools. Table Ill-2 displays the number of treatment 
respondents, by role and state, on the first (2015) and fourth (2018) survey administrations. In 
2018, the online survey was completed by 325 teachers and 67 administrators and other staff in 
the 13 SECEP schools that participated in both survey administrations. The average response 
rate in 2018 was 66% with a range of 50% to 90% in all participating SECEP schools. In 2015, the 
overall response rate was 69%. 


Table IIl-2. Sample Characteristics 


First Survey (2015) Fourth Survey (2018) 
SECEP all Cl MI SECEP all CT MI 

Total Number Total Number Total Number Total Number Total Number Total Number 

(Percentage) of (Percentage) of (Percentage) of (Percentage) of (Percentage) of (Percentage) of 
Role Respondents Respondents Respondents Respondents Respondents Respondents 
Teacher 372 (82.5%) 159 (80.3%) 213 (84.2%) 325 (82.9%) 105 (79.5%) 220 (84.6%) 
Administrator 27 (6.0%) 12 (6.1%) 15 (5.9%) 23 (5.9%) 8 (6.1%) 15 (5.8%) 
Counselor 21 (4.7%) 9 (4.5%) 12 (4.7%) 25 (6.4%) 10 (7.6%) 15 (5.8%) 
Support Staff 20 (4.4%) 13 (6.6%) 7 (2.4%) 13 (3.3%) 7 (5.3%) 6 (2.3%) 
Instructional 3 (0.7%) 3 (1.5%) 0 (0%) 2 (0.4%) 0 (0.0%) 2 (0.8%) 
Coach/Curriculum 
Coordinator 
Other 8 (1.8%) 2 (1.0%) 6 (2.4%) 4 (1.0%) 2 (1.5%) 2 (0.8%) 
Total 451 (100%) 198 (100%) 253 (100%) 392 (100%) 132 (100%) 260 (100%) 


In addition to surveying treatment schools, we asked administrators in 53 comparison schools 
to complete the survey. We identified a total of 53 comparison schools that were similar to 
SECEP treatment schools on various demographic and performance characteristics. Using data 
from the 2012-13 school year, we matched schools on characteristics such as school size, 
poverty percentage, minority percentage, graduation rates, and student test scores. 
Administrators from 28 comparison schools responded to the survey in the spring of 2015 and 
32 comparison schools responded in the spring of 2018. A total of 18 administrators provided 
data at both points in time. The response rate for comparison schools was 51% in 2015 and 60% 
in 2018. 


Staff Survey Analyses Techniques 


In the study, we looked at changes over time for the program schools and then also compared 
the program schools to comparison schools. The analytic approaches differed. 


Program School Analysis Methodology. We looked at changes over time (for SECEP 
schools only) using a mixed-effects analysis of variance (ANOVA). One analytic challenge was 
that, because the survey was anonymous, we could not determine whether the same teachers 
responded at the different time points, making it impossible to link survey results to an 
individual participant across time. Therefore, we were unable to account for the correlation 
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between survey responses across the two administrations, as is typically the case with 
repeated-measures analysis. However, because survey responses were tracked at the school 
level, we were able to control for the fact that respondents were nested within schools. To 
account for the fact that respondents were nested within schools, we used mixed-effects 
ANOVAs for analysis of scales where survey administration (spring 2015 vs. spring 2018) was 
entered into the model as a fixed effect and school ID was entered as a random effect. We used 
contingency table analysis with Chi square tests for items with categorical response options. 
Inferential statistics were calculated for comparisons of every item and scale in the reported 
tables. Statistically significant effects at the item and/or scale level were noted. 


Program and Comparison School Principal Survey Analysis Methodology. For 
SECEP/comparison school analyses, one administrator response per school was selected. In 
order to identify an administrator at each program school for this analysis, when multiple 
administrators completed the survey, we used data from the administrator with the most years 
of service at a program school. In the event that an administrator at a program school did not 
complete a survey, we used data from the school counselor, when available, because 
counselors answered most of the same items that were designated for administrators. 


To compare results between treatment and comparison schools, we used an ANCOVA, where 
item and scale values from the second survey administration served as the dependent variable. 
Item and scale values from the first survey were then entered as covariates in the analysis to 
control for previous levels of the constructs under investigation. For items with binary (yes/no) 
outcomes, we used contingency table analysis with Chi square tests and did not conduct 
SECEP/comparison contrasts. 


Site Visit Methodology 


Site visits were conducted in the spring of 2016 and the spring of 2018. These site visits were 
designed to explore what implementation looked like in practice and identify any lessons 
learned by school staff. 


Sample 


In both 2016 and 2018, the evaluation team visited the same sample of 12 schools in 8 districts 
in Connecticut and Michigan. All schools for the site visits were purposively selected by project 
and ISD staff to be schools that would provide useful information relative to implementation. In 
Bridgeport, the team visited two of the three high schools participating in the grant and two 
middle schools. In each of the four ISDs in Michigan, the team visited one high school and one 
middle school. In Lapeer, Genesee, and Delta-Schoolcraft ISDs, the middle school and the high 
schools were parts of different local districts; in Washtenaw, the high school and middle school 
were in the same local district. In each middle school, the team interviewed the principal and 
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one science and one math teacher. In each high school, the team interviewed the principal, a 
counselor, a group of 6-8 students, and one science and one math teacher. The principal was 
asked to select teachers who had received SECEP professional development or coaching. For 
the student focus groups, the principal selected students who had taken college-level courses. 
Additionally, the team interviewed (1) the person leading the SECEP initiative at the local 
district; (2) the coach (SECEP, STEM, or instructional, if available) working with the schools; and 
(3) a college partner representative. In 2016, a total of 63 adults and 41 students were 
interviewed and in 2018, a total of 61 adults and 33 students were interviewed. 


Measures and Procedures 


The interview protocols were designed to gather participants’ perspectives on the two major 
areas of implementation: (1) the key partner and district-level implementation strategies and 
(2) the SECEP Design Principles. Participants were asked to describe key lessons learned relative 
to implementation. During the first round of site visits (2016), participants were asked about 
short-term impacts of the program. In the second round of site visits (2018), participants were 
asked about: perceived impacts of the program on teachers and students, impacts of the 
program on school culture, their views on sustainability of the project goals and activities, and 
recommendations for the future projects. During the second round of interviews, participants 
were also asked to respond to two mini-surveys, one on perceived impacts of SECEP on school 
culture and a second on the factors affecting sustainability. Participants then commented on 
their responses. A crosswalk with sample interview questions is provided in Appendix B. The 
interviews were recorded and transcribed. 


Analyses 


The evaluation team analyzed the interview transcripts to identify responses relevant to pre- 
specified topics such as sustainability or each of the STEM Early College Design Principles. 
ATLAS.ti software was used to further code themes of responses within each topic. Results 
were then summarized, first at the level of the individual schools, then districts, ISDs, and then 
for Michigan and Bridgeport. Where appropriate, results were further summarized across the 
entire project. 


Additional Interviews 


In addition to the interviews that occurred as part of the site visits, we conducted annual 
interviews with project and district staff that were designed to describe implementation to date 
and capture lessons learned and recommendations. These interviews used the same analysis 
procedures described above. 
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Student Impact Study 


The SECEP evaluation was designed to assess the impact of the SECEP Model on relevant 
student outcomes. The student impact study used a quasi-experimental design in which SECEP 
treatment schools were matched to schools that had similar baseline demographic 
characteristics and performance outcome measures. The study then compared outcomes for 
treatment schools with outcomes for those comparable schools. 


The original intent of the study was to obtain de-identified student-level data from Michigan 
and Connecticut and combine the two sets of data to create an overall program-wide impact 
estimate for SECEP. As will be described in the impact section on Michigan, we did obtain 
access to Michigan student-level data and were able to run most analyses as originally planned. 
However, we were not granted access to student-level or customized school-level data for 
Connecticut; as a result, we needed to use publicly available school-level data that restricted 
our choice of variables and required a different matching and analytic approach. We therefore 
present the student findings as two separate impact studies in two separate report subsections, 
one for Michigan and one for Connecticut. The detailed methodology for each state is reported 
within the relevant section. 
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SECTION IV: IMPLEMENTATION OF THE STEM EARLY COLLEGE DESIGN PRINCIPLES 


The activities described in Section Il were designed to assist schools in implementing the STEM 


Early College Design Principles. This section summarizes findings relative to the four Design 


Principles using data from surveys, interviews, and site visits. 


STEM College-Focused Academic Program (Design Principle #1) 


The first Design Principle has a focus on STEM and consists of multiple components designed to 


create learning environments that prepare students for college and career. The specific 


components—(1) college-level courses, including STEM-focused pathways, (2) activities to 
support college readiness, and (3) improved instructional practice—are summarized below. 


College-Level Courses Including STEM Pathways 


To implement this part of the first Design Principle, districts committed to engaging 90% of 
students in receiving at least one college credit and to creating four- or five- year academic 
programs that included high school and college-level coursework. 


From Year 1 to Year 4, administrators reported significant increases in all measures of 


enrollment in college preparatory and college-level courses and in STEM pathways, with many 
of these increases being detected starting in Year 3. Of particular note, in Year 1, only 2.3% of 
administrators reported that 75% or more of students in their schools were on track to earn 
any college credit. By Year 4 this number had risen to 29%. Similarly, in Year 1, only 7.2% of 
administrators reported that 75% or more of students were enrolled in STEM pathways, but by 
Year 4 this number had risen to 31%. Full results are available upon request. 


Finally, compared to administrators in matched-comparison schools, SECEP administrators 


reported statistically significantly higher percentages of students enrolled in dual enrollment 


courses and in STEM pathways and a higher percentage of students on track to meet minimum 


university admission standards. SECEP administrators also reported higher percentages of 


students on track to meet minimum admission standards for the state university system, but 


this difference was not statistically significant. 


Table IV-1. Student Enrollment in College Preparatory and Dual Enrollment Courses 


This school year, what 


Fourth Survey 


any college credits 
while in high school? 


Administrators 


percentage of your Level of First Survey (2015) (2018) Adj. Impact 
students: Respondent Analysis n Mean SD n Mean SD Change Mean | Estimate 
Are on track to meet Program All Schools 5 
minimum admission Administrators | CT 
standards for the “MI 
state university Program vs. Program 
system? Comparison “Comparison a 
Administrators 
Are on track to earn Program All Schools 43 2.42 0.8 | 41 3.66 


SERVE 
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This school year, what Fourth Survey 


percentage of your Level of First Survey (2015) (2018) Adj. Impact 
students: Respondent Analysis n Mean SD n Mean SD Change Mean | Estimate 
Program vs. (AB COR Fe eye |e 10 lee eerie) |e || epee |e | ee /Beeeeenmetl|e 3-32.08) 0.98* 
Comparison Comparison 18 2.87 
Administrators 
Are on track to earn Program All Schools 
12+ college credits Administrators 
while in high school? 
Program vs. 
Comparison Comparison 18 Pe Al7/ 
Administrators 
Are enrolled ina Program All Schools 42 2.88 11 | 42 3.81 1.3 0.93** -- -- 
pathway or sequence Administrators | CT | 18 | 2.83 | - 10] 16 |. 456 | - is |: 1.73** | - —~ | = | 
of high school and MI 24 2.92 1.2 26 3.35 1.2 0.43 -- os 
college courses Program vs. Program all 3.66 0.95* 
leading to college Comparison “Comparison | 18 | #£| #| #+|  °&4| | °° &'| 271. 
STEM majors? Administrators 


Note. * p< .05; ** p<.01; tp<.10 


During interviews conducted over the course of the project, participants were asked to describe 
their school’s efforts to expand access to college courses for more students and to create STEM 
and other pathways that span high school and college courses. Their responses are summarized 
below, by state. A list of the pathways existing in both states by the end of the grant can be 
found in Appendix C. 


Bridgeport 


Over the course of the project, both the middle and high schools reported undertaking 
activities designed to increase student access to college courses. At the middle schools, these 
efforts focused on improving students’ readiness for high school and, at the high schools, 
efforts focused on expanding access to college-level courses. 


There were efforts at both Bridgeport middle schools to provide algebra courses in the 8" 
grade so that students could take geometry earlier in high school and have opportunities to 
take advanced or college-level math as juniors or seniors. 


At the two Bridgeport high schools we visited, college-level courses were accessed by students 
in a variety of ways, including Advanced Placement (AP) and International Baccalaureate (IB) 
courses, college-level courses taught by high school faculty, and dual enrollment courses taught 
at the partner colleges. Students at one high school described that they could take up to seven 
different AP courses and that the AP and Honors courses were challenging. 


At one high school, college courses were taught by high school teachers certified as adjunct 
faculty through the University of Connecticut. Other courses were taught at partner colleges. 
An administrator described a sense of pride that their teachers were becoming certified, 
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We were very proud of our teachers for taking this step. They are able to attend the 
course over the summer, to receive the certification, and they are here and once they 
receive this certification they are in a position to be able to teach a course. 


A staff member at the same school referenced the growth in enrollments in college courses 
taught at the school by the college-certified faculty, “Last year we had approximately 14 
students who had exposure. This year we had all the way up to 83 (11"- and 12"*-grade) 
students with exposure to these courses.” Another staff member also described the sharp 
increase in number of students taking college courses, 


In the whole state, we had the second highest increase in college courses in our high 
school through UCONN [University of Connecticut]. There was only one other school 
that had a higher percentage than us.... | think that our students are performing well in 
these college-level courses so then, in turn, they want to take more of those courses. 


Although staff reported that the number of courses available to students had increased due to 
certification of adjunct faculty, the number of pathways available to students remained the 
same. In the final year, two high school staff commented that the number of available courses, 
and the number of students taking these courses, should be increased in order to strengthen 
their impact. 


At the second high school, access to college courses had increased through new postsecondary 
partnerships and pathways. By the end of the grant, the school provided access to college 
courses through an Advanced Manufacturing pathway with a local community college to 16 
students per cohort, and through a Health Sciences pathway with a private college affiliated 
with a local medical center, to 12 students per cohort. Three additional pathways were in the 
development stage: Environmental Science, Culinary Arts, and Law Enforcement. 


A participant described seeing students at this second high school beginning to embrace a 
college culture, “Yeah, | see a lot of students wanting to take college classes now” and “I do see 
that the students are coming around more to the idea of college and creating that piece here, 
as you were saying, as a culture.” Despite this, a different interviewee remarked that, while 
access to courses was increasing, it was not yet embraced as a major offering, nor a vision, for 
this school, “I feel like it’s gradually growing but it's not communicated as like, ‘this is our 
vision,’ ‘that's what we're doing.” 


Michigan 


Michigan had a supportive policy environment for expanding access to college courses, 
including legislation that supported both Early and Middle Colleges, which allowed high school 
students to stay fora fifth year to take college courses free of charge. All four of the ISDs in the 
project took advantage of this legislation to either create or expand their Early Colleges over 
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the life of the project. Each of these Early/Middle College Programs had pathways associated 
with them, including pathways leading to Associates degrees and technical career certificates. 


College coursetaking also expanded beyond students in Early/Middle College Programs over the 
course of the project. A principal in one school said, “We have data that show that every single 
year since this project started, our numbers for dual enrollment..., they've increased every year. 
We have more students enrolling in classes.” 


One way schools tried to increase enrollments was by specifically targeting non-traditional 
college-bound students, as was discussed by two individuals we interviewed, “We're really 
working on it this year as we've put more students into college classes who are not your top 
students, more of our average students...” and “It used to be that it was more just your 
university-bound type student. Now, we've reached out and [tried] to encompass more 
students, more of our at-risk population.” 


Districts also tried to broaden access to college courses for more students through a specialized 
“student success course,” which provided students with opportunities to learn about the 
college experience, expectations, and strategies for college success. In two ISDs, the student 
success course was offered for one college credit, and in some schools, it was offered to almost 
all students in a targeted grade level. One stakeholder said, “The biggest impact was the 
opportunity for students to gain college credits. The [student success course], the fact that 


|.” 


everybody leaves with a credit is cool.” According to one student’s description, “The class that 
we took for the college, that's where we learned like how to take notes for our upcoming 
college classes. | feel like that prepared us a little bit for what's going to come in the future.” 
Students in one school who had taken the course described it as “helpful in figuring out what 


they wanted to do.” 


Expanding access to college courses meant some schools faced logistical or financial challenges. 
For example, schools in two districts noted the need to pay attention to finding high school 
staff who could be certified to teach college courses. As one school leader noted, “We began 
the process of looking at our own staff and their skill levels and their abilities and expanding to 
seek adjunct status at the college.” One of the local districts said that they struggled with 
college courses because of the physical distance to their postsecondary partner; as a result, this 
district also sought out online courses and ways to offer the college courses on their own 
campus. 


A core expectation of the grant was to align high school and college courses into pathways. 
Each Michigan district created pathways with the content of those pathways depending on (1) 
the capacities of the schools to provide college content, (2) their alignment to pathways or 
degrees offered at the partner IHE, (3) local career demands, and (4) students’ interest in 
particular career fields. For example, one local district had an Advanced Manufacturing 
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pathway for which students could receive 8 to 10 college credits, a nursing pathway, and was 
also in the process of developing a childcare pathway. In one of the Middle Colleges, the 
pathways were aligned to give students the general education courses needed for an associate 
degree. In this school, all students took the same transferrable courses in their junior year, 
transferable electives in their senior year, and courses in their concentration area during the 
13** year. In another school, the pathways were described as more individualized whereby the 
staff worked with students to identify their career plans and then helped them to develop an 
individual plan or pathway that met those career needs, often through different means (e.g., 
dual enrollment, CTE, or AP courses). The counselor then worked with their college 
counterparts to help each student understand what courses will help them achieve goals 
through high school and into college. 


Although dual enrollment was the emphasis of the grant, AP classes was another mechanism 
for getting college credit. As one district leader commented, “I know the AP classes are not the 
college experience. We understand that, but to us, it's one step in between, and we've never 
offered as much as we have now.” In one of the schools, however, staff noted that AP offerings 
were reduced, because eligible students preferred dual enrollment courses. 


Program-Wide Summary 


In summary, by the final year of the project, districts in both states were working to expand 
access to college courses. In Michigan, each ISD had at least one Early/Middle College, which 
provided extensive access for the students enrolled in that school. Across both states, schools 
had students enrolled in STEM-focused pathways that spanned high school and college often in 
engineering, information technology, medical fields, or advanced manufacturing areas. Schools 
in neither state, however, were close to reaching the SECEP project goal of having 90% of their 
students receiving at least one college credit. 


College Readiness Activities 


Part of the College Ready Academic Program Design Principle includes activities designed to 
increase students’ college readiness, such as their (1) academic preparedness; (2) college 
aspirations and awareness of available options; (3) ability to take responsibility for their own 
learning; (4) knowledge of the college application process and ability to navigate college culture 
and advocate for themselves; and (5) academic college skills (i.e., time management, 
notetaking, organizational and study skills). 


The staff survey looked at schools’ support for college readiness in three different ways: the 
level of College-Going Expectations, the incorporation of College Readiness Goals, and the 
instruction in College Readiness Skills. Results from the survey show significant positive changes 
between the first and final administration of the survey in teachers’ and administrators’ 
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responses on all three scales. In addition, SECEP administrators reported more positive 
responses than administrators in comparison schools on all three scales. However, these 
observed differences between treatment and comparison administrators were only statistically 
significant for the College-Going Expectations and College Readiness Skills scales. 


Table IV-2. College Readiness Scales 


College 
Readiness 
Skills 


STEM Program 
Teachers 


Level of First Survey (2015) Fourth Survey (2018) Adj. Impact 

Scale Respondent Analysis n Mean SD n Mean SD Change Mean | Estimate 
College All Program All Schools 353 2.69 0.7 2.89 0.7 0.19** -- -- 
Readiness Teachers 7 
Goals 

STEM Program 

Teachers 

Program 

Administrators 

Program vs. : 

Comparison Comparison 18 2.74 

Administrators 
College-Going | All Program All Schools 
Expectations Teachers 


Program 
Administrators 


Program vs. 
Comparison 
Administrators 
All Program 
Teachers 


Comparison 


All Schools 


STEM Program 
Teachers 


Program 
Administrators 


Program vs. 
Comparison 
Administrators 


Comparison 


Note. * p< .05; ** p<.01; tp<.10 


For the College Readiness Goals scale, in 2018, staff believed that, on average, it was “mostly 
true” that the school pursued students’ college readiness goals, with 79% of teacher and 86% of 


administrators (respectively) “mostly” or “entirely” agreeing that their school had clearly 


articulated goals for academic college readiness. Teachers reported statistically significant 


increases on three of the four items used in this scale with use of rubrics to measure college 


readiness being the only area evidencing little change. 
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Relative to college-going expectations, on average 75% to 100% of staff “agreed” or “strongly 
agreed” in 2018 that their schools helped students with their postsecondary aspirations and 
plans. There were positive changes across time for all groups with much of this change being 
detected starting in Year 3. On average, in 2018, staff reported working on students’ college 
readiness skills approximately once a week. There were positive changes across time for all 
groups, with significant increases for teachers and administrators. 


Table IV-3 reports results on whether schools conducted activities that encouraged students to 
think of themselves as students who can succeed in a postsecondary institution.’ In 2018, 
approximately 74% of teachers and 88% of administrators reported that their school was trying 
to improve students’ self-image related to their postsecondary academic success. As Table IV-3 
also shows, there were positive increases for all teachers and Michigan administrators between 
2015 and 2018. 


Table IV-3. Activities to Encourage Students to Think of Themselves as Being Able to Succeed ina 
Postsecondary Institution 


First Survey (2015) Fourth Survey (2018) 


Item 


Respondent 


Level of Analysis 


n 


% Yes 


n 


% Yes 


Does your school 
conduct activities 
designed to get all 
students to think of 
themselves as students 
who can succeed in a 
postsecondary 
institution? 


provide supports for 
developing students’ 
college knowledge (that 
is knowing how to 
prepare for college 
academically and 
behaviorally)? 


All Program Teachers 


STEM Program Teachers 


All Schools 


324 


62.0 


293 


74.4 


Program Administrators 


Program vs. : : A 
Comparison Comparison 18 88.9 18 88.9 0.0 
Administrators 

Does your school All Program Teachers All Schools 312 73.1 290 84.1 11.0** 


STEM Program Teachers 


Program Administrators 


Program vs. 
Comparison 
Administrators 


Comparison 


Note. * p< .05; ** p<.01; tp<.10 


2 Note that the table does not report on the statistical significance of the SECEP/comparison contrast because 


Yes/No questions used contingency table analysis with Chi square tests, as described in the Methodology section. 
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Table IV-3 also shows that approximately 84% of teachers and 93% of administrators reported 
that the school provided supports for developing students’ college knowledge. Results show 
positive changes between 2015 and 2018 for all teachers and Michigan administrators. 


During the interviews, participants were asked to describe their school’s efforts over the course 
of the program to improve students’ college readiness skills. In both states, interviewees talked 
about activities to improve students’ (1) academic preparedness; (2) “soft skills,” (i.e., time 
management, notetaking, organizational and study skills) and ability to take responsibility for 
their own learning; (3) college aspirations and awareness of available options; and (4) college 
knowledge (i.e., knowledge of the college application process and ability to navigate college 
culture and advocate for themselves). Their responses are summarized by state. 


Bridgeport 


College readiness activities in Bridgeport middle schools focused primarily on academic 
preparedness and college and career awareness and aspirations. Academic preparedness at 
both middle schools included increasing the rigor of instruction in pre-algebra and algebra in 
the 8" grade because that was identified as an area that was holding students back. At one 
middle school, comprehensive tutoring and after-school academic assistance were 
implemented to help students be successful as they enter high school. 


Students at the two middle schools we visited were introduced to college as early as the 6" 
grade through college visits and use of the Naviance software. As one staff member noted, 


[We’re] trying to get our students ready for college at an earlier stage. Normally, in the 
past, we were just thinking high school, but now we have to think a little bit further. We 
have to have our kids prepared because, to be honest, our students are 
underrepresented at the college level. 


The high schools addressed academic readiness by assessing students’ performance in math 
and English, administering PSAT and SAT tests, offering credit recovery and SAT prep courses, 
after-school tutoring, and incorporating SAT prep into their regular classes. When asked 
whether their school prepared them well for college academically, students had mixed 
responses. Some believed that their regular classes did not prepare them well, while AP classes 
were more rigorous and prepared them for college-level classes. As one student commented, 
“I'd say the school didn't so much academically prepare me for college, as it did more ona 
social level, like a more mental level. Like, it's helped with my maturity and stuff like that.” 


College aspiration activities at the two high schools included career and college exploration 
courses and activities, college recruitment visits to high schools, and campus visits by students. 
Staff in both Bridgeport high schools described the use of the Naviance career guidance 
software to prepare students to be successful in high school and post-secondary work or 
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school. The effort made at the 9'-grade level in both schools was to help students adjust to 
high school, develop good academic habits, and, as one staff member put it, “start that 
[readiness] conversation.” The focus was on 


...what does it mean to be a high school student, what kind of expectations do we have 
and exposing [students] to some information of what do they need to graduate and how 
it's going to help them most in terms of being college ready. 


The goal was to show students how to make better decisions about college and what behaviors 
are necessary to be college ready. 


Both high schools introduced a course focused on academic college skills (sometimes referred 
to as soft skills) and college knowledge. At one school, the freshman seminar course was 
delivered by faculty from the local community college and was designed to familiarize 9*"- and 
10'-grade students with course syllabi, time management, and self-advocacy. Twenty-nine (29) 
students took part in this freshman seminar in 2017-18. At the other high school, the Career 
and College Readiness class was designed to be taken by 11"- and 12'*-grade students during 
the first period of the day. Teachers from several content areas provided instruction for this 
class; it focused on “...time management skills, the basic responsibilities about how to become 
a student,” and counselors would then “work content into the class.” 


Teachers remarked that they have seen changes in students who have come through the 
college readiness activities in the last few years. Students who had never thought they would 
attend college, now think of themselves as college-going, as one teacher stated, “Some of my 
first-year students that are now seniors, say things like, ‘I didn't think | was going to go to 
college and | got into college.’" One stakeholder also described the changes in students, 


They are feeling that they can accomplish and achieve more than maybe they thought 
they could when they started the year, or came here, or whenever. | noticed that they 
take more pride in their work. | have more students who want to finish. | have students 
who say, "I didn't quite get it all done" instead of feeling that every class problem or 
work they're working on is a grade then-and-there, and punitive. 


Michigan 


Similar to Bridgeport, college readiness activities in Michigan middle schools focused primarily 
on academic preparedness and college and career awareness and aspirations. Stakeholders in 
all four ISDs shared that, in middle school, many students performed poorly, especially in math, 
and thus entered high school unprepared. In three of the four ISDs, there were efforts to bring 
students’ math skills up so that some of them could take Algebra | courses in 8" grade, which 
would allow them to access college courses in later grades. 


With regard to college aspirations activities, staff at two of the four ISDs described 
opportunities for middle school students to visit colleges. Other schools had aspirational 
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activities that included college fairs or college nights, alumni or community speakers, and 
“college days.” One interviewee described activities at the middle school, saying, “We have 
‘wear your jersey day’ during football season, and staff and faculty make it known where they 
went to college, and why they chose to go to school there.” Another staff member described 
daily conversations, “We don't say, ‘Are you going to college?’, we say ‘What college are you 
going to?’ So, college is talked about quite a bit.” 


Across all four ISDs, students began developing their plans for high school and beyond starting 
in middle school. Stakeholders at two ISDs described 8"-grade students working with staff to 
develop their Educational Development Plan. Students used and updated these plans for career 
and college exploration throughout high school. One middle school offered a class for college 
and career readiness for their 8'*-grade students. In another school, college and career 
exploration was part of a daily advisory program. 


At the high school level, schools focused on developing different aspects of students’ college 
readiness. One ISD was intensely focused on high school students’ academic preparation. They 
increased the frequency and number of students being tested in reading and writing and 
prepared more students for Accuplacer. They offered a remediation college lab, various after- 
school academic programs, and increased the number of students in their summer bridge 
program to prepare students for college-level courses both academically and socially. 


In three ISDs, schools offered a college readiness seminar (for one college credit in two of these 
three ISDs). The seminar provided instruction on academic college skills, college expectations, 
and navigating college procedures. One counselor at a different ISD described a change in 
students who have participated in the seminar, 


It causes students to do some thinking...about their future and [we’ve] had the 
compliment many times from our counselors that students in 10* grade now are 
starting to come down and ask some really good questions about their futures and 
careers very early. [Students are] starting to get concerned about these things and they 
do other things as well in the course like looking at the differences between high school 
and college. 


In one ISD, a participant described that teachers embedded career exploration in their regular 
subject classes, while in another ISD, teachers described embedding soft skills into their 
curriculum and grading college readiness as a separate skill, 


We've added an extra column to our grade book that it's basically a “college readiness” 
column so that teachers are able, [for] every single student that teacher has, [to] go in 
and put in a grade. It's a letter grade just saying whether or not they're college ready, 
they're moving to be college ready, or they're not quite there yet, as well as notes on 
what they can work on, social skills, turning in assignments on time, and stuff like that. 
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Program-Wide Summary 


In summary, all middle and high schools in both states focused on academic preparedness and 
college and career awareness and aspirations. Additionally, high schools in both states offered a 
variation of a college readiness course, focused on academic college skills (“soft skills”) and 
college knowledge. Some schools taught these skills in a separate class whereas others 
embedded them in regular subject classes. Students in both states found these classes helpful. 


Improved Instructional Practice 


SECEP also had a strong emphasis on improving instruction to prepare students for success in 
college and STEM fields. Table IV-4 shows the results of two survey scales characterizing 
Rigorous Instructional Practices and Inquiry/Project-Based Learning Practices. Results show 
overall positive, but not always statistically significant, changes over time on both scales. STEM 
teachers reported larger changes in both instructional practices than all teachers combined. By 
the final survey administration, teachers in both states reported using Rigorous Instructional 
Practices, on average, once or twice per week and Inquiry/Project-Based Learning Practices, on 
average, once or twice per month. On both scales, SECEP administrators reported a higher 
frequency of teachers’ use of these practices than did teachers themselves. Finally, most of the 
statistically significant changes in both scales over time were detected during the final year of 
the survey, suggesting that bringing about instructional changes was a slow process. 


Table IV-4. Instructional Scales 


Level of First Survey (2015) Fourth Survey (2018) Adj. Impact 
Scale Respondent Analysis N Mean SD n Mean SD Change | Mean | Estimate 
Rigorous All Program All Schools 3.83 0.8 0.8 
Instruction Teachers i ~ ' “07 | 014 | —- | -— | 
STEM Program 
Teachers 
Program 
Administrators 
Program vs. 5 
Comparison Comparison 18 3.54 
Administrators 
Inquiry/ All Program All Schools 340 2.99 0.7 306 3.14 0.7 0.16** 
Project-Based Teachers 7 ~ ‘ “os | 015 | —- | -— | 
Learning MI 198 | 2.93 | 0.7 | 212 | 3.21 | 07 | o1s** | - | 
STEM Program All Schools 146 2.84 0.7 140 3.11 0.8 0.27** 
Teachers ~ ‘i “o9 | o35* | — | -— | 
MI 2.86 | 0.7 | 100 | 3.09 | 07 0.23* - afl 
Program 
Administrators 
Program vs. é 
Comparison Comparison 18 3.43 
Administrators 


Note. * p< .05; ** p<.01; tp<.10 
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During interviews, participants were asked to describe their school’s efforts over the course of 
the project to change STEM instruction, and instruction in other subjects. Their responses are 
summarized separately for each state. 


Bridgeport 


Early in the project, Bridgeport brought focus to their STEM curriculum by pushing for greater 
technology integration into math and science classrooms. SECEP district leaders identified a 
cohort of math and science teachers from all of the SECEP high schools to receive professional 


development from Teachers 
College instructors, some of which 
was about integrating technology 
into the science classrooms and 
some about project-based 
learning strategies. 


Over the life of the project, the 
district also focused on 
implementation of the Common 
Instructional Framework (CIF), six 
instructional strategies designed 
to increase students’ engagement 
in learning (see box). By the time 
of the interviews with district and 
school staff in Year 4, 15 
participants reported that 
instruction had changed at the 
middle and high schools in 
Bridgeport as a result of the 
project. All seven interviewed 
teachers described their increased 
use of CIF strategies in their 
classrooms. Three teachers 


The Six Strategies of the Common Instructional 
Framework (Jobs for the Future, 2012) 


Collaborative Group Work: Collaborative Group 
Work brings students together in small groups for 
the common purpose of engaging in learning. 


Writing to Learn: Writing to Learn enables 
students to experiment every day with written 
language and to increase their fluency and 
mastery of written conventions. 


Scaffolding: Scaffolding helps students connect 
prior knowledge and experience with new 
information and ideas. 


Questioning: Questioning challenges students and 
teachers to use good questions as a way to open 
conversations and further intellectual inquiry. 


Classroom Talk: Classroom Talk creates the space 
for students to articulate their thinking and 
strengthen their voices. 


Literacy Groups: Literacy Groups provide students 
with a collaborative structure for understanding a 
variety of texts, problem sets, and documents by 
engaging in a high level of discourse. 


specifically described how they changed their instruction to include increased use of 
collaborative groups, classroom talk, and writing to learn. Teachers stated that students were 
more engaged as CIF strategies became part of the classroom and that students were able to 
learn more in depth, as one teacher said, “I see a student that understood something and now 
is able to explain it maybe at a simpler level. That’s challenging. You understand something 
that’s complicated, and you can explain it at a simple level, that’s intelligence.” 
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Three teachers described how they incorporated more authentic problems into their 
curriculum. For example, in one STEM class, students researched careers that they were 
interested in and then were assigned a project that required them to explore that career and 
the math that was required in that field. The teacher described one student’s revelation about 
how her chosen career actually required math, 


[Initially she said] “I'm going to be a hairdresser, | don't need to know math.” And 
there's a whole math of creating the hair dye and that was new to me and new to her so 
then she became more interested in the math, “Oh, well | need to know about ratios 
when | mix the colors and things like that." 


Three teachers reported collaborating with colleagues to develop cross-curricular lessons that 
would apply CIF strategies to STEM projects. The teachers have incorporated Collaborative 
Group Work, Scaffolding, and Writing to Learn to engage students. 


Although the interviewed teacher leaders reported implementing the CIF strategies with 
confidence, one district leader, who often observed teachers, described CIF implementation 
across the schools as “uneven.” One stakeholder shared that this uneven adoption may have 
been due to the fact that teachers lacked understanding that the strategies were to be 
embedded in regular classroom practice, stating, “They see it as add-on that they may or may 
not have time to do.” According to a district leader, teachers who implemented the CIF 
practices effectively modified their lessons as they gained experience, and thus, they modeled 
persistence for their students. One teacher leader was observed working with students during 
the project. The observer described behaviors of the students in the class where the CIF had 
been implemented, 


I've observed students in those classes where teachers have taken hold of it and they 
make it work and they question it, and they figure out what didn't work, and they try 
something else. | see their students doing that [same thing]. They're seeing that you just 
don't walk away from things. The students start to think, "I just don't walk away from 
this." 


While teachers in all schools described adopting CIF instructional strategies, students in both 
Bridgeport high schools did not always witness their implementation. In one school, the 
primary challenge was teacher turnover. Students described some good teachers but noted 
that most of these teachers had left the school. Students also described poor experiences in 
some classes as compared to others. For example, in another high school, students viewed their 
science classes as taught in much more engaging ways than their math classes. They described 
little interaction and a lack of CIF strategies being used in these math classes. 
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These results suggest that instructional change relative to the CIF and STEM-focused inquiry 
activities occurred with some teachers and some classrooms but that by the end of the grant 
instructional changes may not have been as widespread as desired. 


Michigan 


In Michigan, efforts to improve instruction were brought about by focusing primarily on 
project-based learning. Across all schools, teachers indicated that there had been increased 
development and use of projects over time with a middle school teacher describing how cross- 
curricular projects were implemented in their school, 


Each grade has at least one project...we have students that have geometry and biology, 
and they are working on building a spaghetti bridge. In science they are talking about 
the environmental impact of a bridge and in math we are building a bridge to see whose 
bridge will hold the most weight and then we're going to calculate the forces. 


A middle school in another ISD made substantial changes to the master schedule so that all 7*- 
and 8'-grade students spent the final two periods of the day (100 minutes) in project-based 
learning (PBL). Before lunch, teachers incorporated STEM and projects into their classes as they 
liked. In the afternoon, which was required by the administration to be “100% PBL,” all teachers 
led groups of up to 18 students in six-week-long inquiry projects. One teacher shared that this 
PBL was the “biggest change” in the school and said that it allowed teachers to work with 
smaller groups of students. 


Similarly, staff in the high schools reported making changes in their instruction, also focusing 
more on problem- or project-based learning and inquiry. A counselor described how the science 
class was using more PBL, 


When | walk by [the science class] they're so engaged with their problem-based 
learning, and it was just cool to see because | don't think they are kids that maybe 
would have been as engaged in science and in math. 


A high school student in the same school agreed, describing how a project helped them develop 
an understanding of physics concepts, 


We just did a chapter on simple machines, and we used levers and pulleys, and we're 
figuring out the force and how much work you end up actually doing with the use of 
those simple machines. And that really helped [us] understand like, yeah, you're doing 
more work, but the work is easier [because of the simple machine]. 


Because most of the projects were cross-curricular, development involved collaboration among 
faculty. In one district, this collaboration even occurred across schools with the middle and high 
school staff working together on projects. An interviewee noted that this was unusual, 
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We've rarely had cross-building project work. We have done some cross-building 
projects where teachers have given opinions but not really influenced the work. We've 
had our STEM integration team, over these last several years, which has actually had 
middle school teachers sitting in a room with high school people developing activities 
for high school students; that kind of collaboration rarely happens and vice versa. 


Challenges to incorporating projects on a large scale were expressed by teachers at three 
schools. One teacher described large variations in student knowledge and skills (e.g., 
accelerated math students grouped with students who are below-grade level) and how this 
disparity can be frustrating for teachers who are leading the groups and trying to accommodate 
all learners and frustrating for students who feel held back or who could not keep up. However, 
another teacher believed that projects helped students at different learning levels: “I think 
that...having nice open tasks gives an entry point for all students. Then that means that even 
struggling students have that opportunity to kind of interject something. They have something 
that they can say.” Other teachers recognized that problem- or project-based learning required 
scaffolding. One teacher described guiding the class, “‘here is what we know’, ‘this is what we 
want to know’, ‘how do we get there?’...ask questions, develop ideas, etc.” 


Another challenge noted in one school was that the school’s structure made doing cross- 
subject projects difficult because students were not in “houses” and did not share the same 
teachers. Finally, staff in one school noted that implementation of projects had been on hiatus 
during the 2017-18 school year because of leadership turnover; however, in the hiring of a new 
principal, the expectation was that projects would be implemented in the upcoming academic 
year. 


Across the Michigan schools and districts, there were a variety of STEM-oriented extracurricular 
experiences for students, the most common being robotics, which was offered in all of the 
districts. In one district, the robotics club started in middle school and continued into high 
school. One district highlighted participation in math and science competitions, including 
Science Olympiad. One district provided engineering opportunities through Project Lead the 
Way. One middle school had introduced the I|QWST curriculum from Michigan State University, 
although that has led to the loss of some technology and robotics electives. 


Program-Wide Summary 


Across both states, there have been reported changes in instruction in all schools. In 
Bridgeport, schools implemented the CIF strategies, and in Michigan, schools focused on 
implementing project-based and inquiry learning. While all teachers we interviewed 
implemented new instructional strategies, the survey results suggest that not all teachers in the 
schools had embraced the changes. 
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Student Support (Design Principle #2) 


The Student Support Design Principle recognizes that an increasingly rigorous academic 
environment requires extensive academic and affective supports. As articulated by the model, 
these supports are expected to be collaborative with the college. Schools are also expected to 
support students’ development of “college knowledge,” including assistance in college and 
career planning and in the transition to college. This section describes the survey and interview 
results around academic and social-emotional supports, including the support provided to 
students around the college application process. 


Before SECEP began, schools already reported high levels of supports in place. Across the life of 
the project, schools and districts increased their support in areas related to college readiness 
and students’ performance in college classes. Year 3 data revealed that, compared to other 
Design Principles, supports for students in college-level courses lagged and had just started to 
gain more attention from district leaders. During that year, two districts started receiving more 
information from colleges about the progress of their students in college classes and as such, 
districts began providing supports for these students on high school campuses. 


Table IV-5 reports on two different support scales from the staff survey: (1) whether there were 
Schoolwide Systems of Support in place, and (2) whether staff provided supports themselves. 
Results from the implementation survey showed inconsistent change across years on these 
scales. The survey showed that staff reported an overall increase in school-wide system to 
Support students. There was also an overall increase in the extent to which school staff 
provided academic and social supports themselves, however, this was driven by Connecticut 
staff only, which was likely the result of an unrelated social-emotional program (see description 
in sub-section on Connecticut). Reported supports on both scales were higher in SECEP schools 
than in comparison schools. 


Table IV-5. Student Support Scales 


Level of First Survey (2015) | Fourth Survey (2018) Impact 
Scale Respondent Analysis n Mean | SD n Mean SD | Change Estimate 
School System for | All Program All Schools 2.75 0.6 3.04 0.6 
Academic/ Social- | Teachers i sas fesesectcceeenenensctecetensneceetnfecttnneceeettaneeenpeceeetneeeeettaeeeeetaae 
Emotional  =s—CisdN( ak ETO RE To I0 178 Ga 1 ey ae ne 
Support STEM Program 
Teachers §|cl | 57 | 266 |05 1 39 | 301.|07 1034%* | ~ | «~~ 
Program 


Administrators 


| Programvs. +*| Program | 12| + #2»| + +~|  +|  &+2|  +| | 323 | O37 | 

Comparison Comparison 18 2.86 

Administrators 
Staff Provision of All Program All Schools 330 3.31 0.5 | 304 3.43 0.5 0.11** -- -- 
Academic/ Social- | Teachers ' ai) i 
Emotional MI 3.36 | 0.5 3.41 | 06 00, | 
Support STEM Program All Schools 143 3.29 0.5 3.36 0.5 0.07 -- -- 

Teachers : “il r /: 
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Scale 


Level of First Survey (2015) Fourth Survey (2018) Adj. Impact 
Respondent Analysis n Mean | SD n Mean SD | Change | Mean Estimate 
MI 3.36 
Program 3.45 | 


Administrators 


Program vs. 
Comparison 
Administrators 


Comparison 


Note. * p< .05; ** p<.01; tp<.10 


The survey also included two questions about the percentage of students who received 
assistance with the college application process (Table IV-6). Between 2015 and 2018 there was 
an increase in the percentage of students who were aided in applying to college and for 


financial aid in Michigan and a smaller change in Connecticut. In 2018, more than 90% of 


students, on average, were provided with such assistance. SECEP schools did not significantly 


differ from comparison schools on the number of students provided with such assistance. 


Table IV-6. College Application Assistance 


First Survey 


grade, what 
percentage of 


Administrators 


Level of (2015) Fourth Survey (2018) Adj. Impact 
Administrator Item Respondent Analysis n SD n Mean | SD | Change | Mean | Estimate 
By the end of 12" Program All Schools 3.71 0.5 


applying for 
financial aid? 


Administrators 


students are given Program vs. d 

assistance in Comparison Comparison 18 3.65 

applying to college Administrators 

(e.g., selecting a 

college, completing 

applications, etc.)? 

By the end of 12 Program All Schools 39 3.21 0.8 42 3.69 0.7 | 0.49** 

grade, what Administrators | CT | 16 | 3.19 | 08 | 16 | 3.56 |o6| 038 | #&2x| 
percentage of MI 23 3.22 0.9 26 3.77 0.8 | 0.55** 

students are given Program vs. Program 12 3.85 0.25 
assistance in Comparison “Comparison |; AT [Pe ay eee Poem el a ol ey Mt aie ACT hg 


Note. The response scale was “less than 25%” (1), “26-50%” (2), “51-90%” (3), and “91-100%” (4). * p< .05; ** p< .01; tp<.10 


During the interviews, participants were asked to describe their school’s efforts to expand 
student support because of SECEP. In both Bridgeport and Michigan, supports were provided by 
teachers, district/ISD staff, and the college partners, although it was not always clear that 


changes in supports were due to SECEP. Support ranged from adding staff (like a district liaison) 


to changing school schedules to include advisory times or seminar classes. The interview 
findings mirrored the survey in that, from the onset of the grant, both Bridgeport and Michigan 
were focused on providing students with supports to prepare and support them in college 


courses. Interview responses are summarized below. 
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Bridgeport 


Over the course of the grant, Bridgeport expanded the supports received by students, some of 
which were due to the grant and some due to other projects. 


Academic Supports 


Both the district and the community college had staff members dedicated to helping students 
bridge the gap between high school and college. As part of SECEP, the district assigned the 
college liaison role to personnel who provided supports and monitored students taking college 
course. The college liaison initially worked with a group of 100 students to prepare them for the 
college entrance test (Accuplacer). Students who passed this test could then begin taking 
college courses. In Year 2, the district college liaison continued to provide support for students 
enrolled in college courses. In the past, eligible students enrolled in college courses did not 
receive enough support and often dropped out of these courses. As a district staff member 
said, “...I think the college liaison has been the game-changer for us, that level of support.” 


During Year 1, the district also planned a new college readiness course for 9" graders that 
would focus on study, organizational, and social skills; they also scheduled college tours for high 
school students. The SECEP grant also supported math interventionists who worked with 
underprepared students in class or during pull-out times to get them on the grade level 
academically. The math interventionists also shared their instructional strategies and other 
tools with the classroom teachers. 


Schools also initiated their own supports for students. For example, during Year 4, all staff that 
we interviewed at a middle school discussed using Scientific Research-Based Interventions in 
the afternoons, when students were organized into small groups based on academic need. At 
the high school level, respondents reported numerous academic supports, including the 
Reading Plus Program for English, STAR Renaissance Program, Khan Academy, and test 
preparation programs (e.g., SAT prep). A principal at one school and a teacher at another school 
mentioned using lab classes to help struggling students. Also, a teacher at one high school 
mentioned a new after-school tutoring program that was started during Year 3. 


Socio-Emotional Support 


In terms of social and emotional support, middle school staff, high school staff, and district 
representatives reported that schools used Yale University’s RULER Program 
(http://ei.yale.edu/ruler/how-ruler-works/) as well as a restorative justice program, both of 
which were not supported by SECEP but were seen as complements to the program. 
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College Knowledge 


One of the areas in which the grant appeared to influence student support was relative to the 
development of college knowledge. After SECEP began, the schools started offering activities 
designed to build students’ college knowledge. For example, the partnership with the 
community college led to a high school coordinator going to high schools to help students 
complete their applications and attended high school meetings. Throughout the years, the 
college liaison helped students to use Naviance software to develop college readiness 
preparedness. 


Both the district and schools implemented different initiatives to ensure students were 
knowledgeable about college. For example, during Year 2, the district held a college and career 
awareness week to increase college-going culture, and a college and career readiness day, 
when all high school students took the SAT or PSAT, and all 8'" graders took the ReadiStep test. 
One of the high schools also developed a college skills curriculum that oriented students to the 
first year of college; this curriculum was put in place for all 9" graders. 


Although staff at the middle and high schools, as well as district representatives, reported a 
number of efforts, some noted that formal supports were lacking and that this was due to 
limited resources at the schools. As one teacher said, 


| don't think that there's really a clear direction of promoting, like, having that 
academic support, having that social support, encouraging kids to go to college, 
helping them get to college, giving them the resources and skills that they need. | 
think that's something that needs a lot of improvement here. 


In sum, the new supports provided by SECEP were the college liaison, the math interventionists, 
and the college readiness curriculum; other supports were either in place prior to the 
implementation of SECEP or were associated with programs not affiliated with SECEP (e.g., 
RULER, restorative justice). 


Michigan 


Our interviews with school, district, and college partner staff suggest that academic and college 
knowledge supports for students varied within, and across districts/ISDs. 


Academic Supports 


When SECEP began, the ISDs immediately started planning supports for students to be provided 
by schools or college partners. By the end of the project, five of the eight site visit schools had 
formally embedded, to some degree, academic and social supports into the school schedule. 
This was accomplished via seminar classes, advisory times, or freshman academies focused on 
topics ranging from academic support, soft-skills development, and/or college readiness 
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support. In two schools in one ISD, school leaders acknowledged the role SECEP played in the 
development of specific supports embedded in the curriculum. For example, in one 6-12 school, 
support time was embedded into a project-based learning class so that students could get 
academic support or help with homework. The other high school that we visited in the district 
offered a First Year Experience course to help less-prepared students develop the skills needed 
for future dual enrollment courses. 


In some cases, ISDs and college partners collaborated to provide the necessary support. For 
example, one district, in partnership with the local Early College, provided a summer bridge 
program that focused on college readiness in reading and writing. Another district worked with 
their postsecondary partner, which has a required course for freshmen college students 
focusing on college readiness and study skills. The district schools integrated this course’s 
elements into some of the required courses for high school freshmen, such as English. 


In Year 3, two ISD coordinators described how their college partners had started to share 
information about dual enrolled students with the schools, especially struggling students. This 
information exchange allowed schools to intervene before these students failed their college 
classes. One of the coordinators indicated that this more proactive support by the college 
represented a shift in perspective, 


The college's directors are contacting the high school staff if a kid is struggling and then 
the high school staff is following up with that kid more quickly. It used to be, even with 
the dual-enrolled class on campus, they still viewed the kid as an adult, you know, this is 
up to you. We don't want them to fail.... Four or five years ago they [the college] 
wouldn't do that. 


Affective Supports 


Supports in Michigan tended to focus on academic and college readiness. Other than support 
around soft skills (as described below), no interviewees described any emphasis on social- 
emotional supports. 


College Knowledge 


Early in the project, project partners in Michigan asked participating ISDs and local districts to 
map the college knowledge necessary for students to enter and succeed in college. The districts 
thus had a strong focus on preparing students for college in their academic and soft skills, such 
as study skills. All districts worked to develop courses or curricula focused on college readiness 
as described under the STEM College Ready Academic Program. 


One district developed an explicit college readiness curriculum. They also received a community 
grant to pilot a peer advisory model which used students with college experience as advisors to 
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peers under the guidance of counselors. The local high school encouraged and prepared 
students to take and pass the COMPASS assessment, do the online college application, and take 
college tours. 


Although increased efforts to support students was evident, interviewees commented on the 
somewhat informal nature of the supports and expressed concerns that students were not 
utilizing offered supports. For example, in Year 4, in three schools across two ISDs, multiple 
respondents reported heavy reliance on informal academic and social supports provided by 
counselors and teachers. A teacher noted that this informality made it difficult to know “who’s 
going to make the first move” in terms of identifying and supporting struggling students. Two 
staff members in one school attributed this lack of formal supports to staff turnover. 


There was a concern across all four ISDs that while the community colleges offered a number of 
academic and social supports to students, dual enrolled students were either underutilizing 
these services or were unaware of their existence. As one college representative noted, dual 
enrolled students “are not fully aware of the supports that the community college has to offer, 
the tutoring labs and all of the different resources.” 


Program-Wide Summary 


In both Bridgeport and Michigan, students were provided academic, affective, and 
postsecondary support. While academic and postsecondary support appeared to be more 
formal, there seems to have been a less systematic approach for the affective support (except 
for the RULER program in Bridgeport). In both Bridgeport and Michigan there appeared to be 
an increase in supports from the postsecondary partnerships; it ranged from preparing students 
for college to supporting them while they took their dual enrollment courses. In both sites, 
these increased supports were provided by the district administrators and college partners. 
However, the interviews indicated that some of the increased supports were not a part of 
SECEP. 


High School-College Collaboration (Design Principle #3) 


The third Design Principle, High School-College Collaboration, involves developing strong 
partnerships between the districts and postsecondary institutions. Evidence of these 
partnerships includes the signing of a formal MOU between the district and at least one 
postsecondary partner, establishing at least one career/college pathway in each school 
annually, sharing of resources, and ongoing communications between districts and colleges. 


A theme that emerged across all data collected over the course of the evaluation was that 
districts and college partners strengthened their relationship over time and that, in many cases, 
SECEP helped facilitate new collaborations in addition to supporting preexisting relationships. 
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High School-College Collaborations were explored via two scales on the staff survey—one that 
measured number of times high school staff collaborated with college faculty (High School- 
College Collaboration Scale) and one that measured the supports provided by the college 
partner (Local College Support Scale). Results from staff surveys showed that, in general, there 
were positive changes between the first and final implementation of the staff survey, on both 
scales (see Table IV-7). Also, SECEP administrators reported higher levels of support from their 
local college than administrators at matched-comparison schools who were not part of SECEP. 


Table IV-7. High School-College Collaboration Scales 


Level of First Survey (2015) Fourth Survey (2018) Adj. Impact 
Scale Respondent Analysis n Mean SD n Mean SD Change Mean | Estimate 
High School- All Program All Schools 1.32 0.6 1.45 0.7 
College Teachers [ ” ; T 


Collaboration 


STEM Program 
Teachers 


Program 
Administrators 


Program vs. 
Comparison Comparison N/A 
Administrators 
Local College All Program All Schools 294 1.92 0.7 285 2.16 0.8 0.24** 
Support Teachers i i ie “a 


STEM Program All Schools 129 | 1.84 0.6 128 | 2.14 | 08 0.29** 
Teachers 


Program 
Administrators 


Program vs. 
Comparison Comparison 18 1.85 
Administrators 

Note. * p< .05; ** p<.01; tp<.10 


The interviews supported these survey findings. Our initial interviews with district and school 
staff indicated that most districts collaborated with postsecondary partners prior to SECEP. 
However, a consistent finding from our interviews was that, over time, existing relationships 
were strengthened and new relationships formed as districts and college partners worked 
together to expand and improve dual enrollment and pathway opportunities. Additionally, 
several respondents in Bridgeport and across the Michigan ISDs, particularly the smaller ISDs, 
indicated that many of the new and strengthened relationships came about as a result of the 
district’s involvement in SECEP. In the remainder of this section we provide a summary of what 
we learned from our interviews with representatives from Bridgeport and the Michigan 
districts. 
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Bridgeport 


In the first year of the project, Bridgeport had letters of agreement in place with a local 
community college, the local state university, and three local private universities. The three 
high schools were working to create four career pathways for students. Bridgeport also 
modified its master schedule for 9*- and 10*-grade students to better prepare students for 
community college courses. As noted earlier, Bridgeport also hired a district college liaison to 
help students take and pass college entrance exams to improve enrollments in college courses. 


Over time, collaborations strengthened, and new ones were developed. For example, in the 

second year, there was a new collaboration with a local university that resulted in a program 
where 80 students at one high school were bussed to the university for an entire day, every 

other week, to take high school science and math classes on the college campus. In the third 
year, one of the high schools started a new partnership with a local hospital, which provided 
students at that school with an opportunity to take part in a health-related pathway. 


By the time of our final interviews, district staff and college partners reported a number of 
activities where schools, districts, and college partners actively worked to improve the 
partnership and expand opportunities for students. For example, one college partner discussed 
a new initiative around hosting a district counselors’ meeting to familiarize the counselors with 
the work of the college and to bring about greater awareness of opportunities for students. The 
college representative emphasized the importance of such activities by saying, 


It's important that the college knows the district and that the district knows the 
college, and not just president and superintendent, but down the line. And it's 
also important that not only each knows what the value of resources are in each 
respective entity but also what's going on. If we really want to exploit all that 
both has to offer, we have to have synergy of what's going on in terms of the 
different programs and activities.... So, | think that this programming allows 
for...use [of] that lens to bring those things into focus. 


At both high schools and one of the middle schools visited in Year 4, respondents reported that 
college partners were engaged in a number of collaborations with school staff as part of, and in 
addition to, dual enrollment and providing access to high school-college career-themed 
pathways. In general, college partners have (1) provided information sessions to students and 
parents around college program availability, (2) hosted student field trips to the college, (3) 
worked with students and parents with the application process, and/or (4) facilitated 
information sessions around topics such as completing FAFSA applications. 


All interviewees who mentioned college partnerships spoke of these partnerships as being 
positive and, in many cases, relationships were deepening or growing “exponentially.” Although 


BqQseRveE 38 


Center 


many of the partnerships were in place prior to the grant, four respondents discussed how 
SECEP helped strengthen these relationships. 


Michigan 


In Michigan, formal MoUs between districts and postsecondary partners were not developed 
until the second year, primarily because the MoUs had to address the challenging issue of how 
much each respective entity would pay for the costs of the college courses. Despite the lack of 
formal MoUs, schools in Michigan started creating pathways in collaboration with community 
college partners in areas such as engineering, construction technology, and various medical 
fields. 


As with Bridgeport, stakeholders reported that relationships between the districts and 
postsecondary partners grew stronger over time. In the two smaller ISDs, each of which had 
only one community college partner, district representatives reported that these collaborations 
started as a result of SECEP. As one ISD coordinator put it, “The college was completely 
separate from the K-12 school systems and now we’re working together, and we have a 
common goal, and we’re collaborating much more.” The two larger ISDs had a number of 
college partners in place prior to the grant, but the partnerships were strengthened as a result 
of the grant. 


Although collaboration improved across all districts, the quality of collaborations varied across 
ISDs. For example, in one district, the college did not participate in regular project meetings. In 
another district, however, a respondent described very active participation, 


We have a designated college liaison. He is a science instructor over at the college. We 
also have a member of the college who is the Dean of Arts and Science and he has been 
participating in every SECEP meeting and he’s been an integral part of bringing the 
college into the high school.... [with] their staff working with the high school staff and 
talking to them about aligning the curriculums in both high school and college. 


The primary activity of the collaborations was expanding access to college courses, but colleges 
also provided support to districts in various ways. For example, in one ISD, the college partner 
helped the whole district science department to learn what it means to prepare their students 
for college-level courses. One coordinator described the role of the postsecondary partner, 


[Staff from the college] go to the local schools. They've done financial aid workshops for 
the general populations. They've been there for parent meetings and student 
assemblies when we want to talk about all these options. Kids in the 10° grade get 
bombarded with options in their junior and senior year. 
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One ISD did experience a setback with a college partner due to a leadership change at the 
college, combined with opposition from district and college unions. However, even in this 
district, the relationship improved such that the college was, for the first time, sending college 
instructors to the high school campus and reducing their fees to help the schools and districts 
meet budget constraints. 


By the final year of the project, relationships had improved such that schools, districts, and 
college partners were working together to solve logistical issues around rising dual enrollments 
or creating new pathways. This led staff from both organizations to form closer working 
relationships and learn more about each other’s organization. As one district leader said, 


| think now that we've been in this for four years, the colleges really understand, 
especially in a school like ours, all of the components that we have to deal with, 
and all of the challenges that our students bring to us. And there's a lot less 
blame game on their part, when the students go over there, because they get it 
now. And also, we understand the system that they work in, and who they have 
to report to with the board and their board’s expectations. It's really more of a 
shared understanding and we're much...quicker to accommodate based on 
understanding what they need and what we need. It's a much nicer flow of 
information back and forth. 


Program-Wide Summary 


In both Bridgeport and Michigan, it was evident that the grant had increased the number of 
postsecondary partners and strengthened ongoing partnerships. Across both states, there was 
a clear expectation that this would be one of the most sustainable parts of the grant. A staff 
member in Michigan said, “It is going to be a great partnership because we're so intertwined.” 


Culture of Continuous Improvement (Design Principle #4) 


The final Design Principle involved the creation of a Culture of Continuous Improvement 
through regular reflection around and use of high school and college performance data, 
ongoing teacher collaboration and professional development, and alignment of high school 
curriculum with college expectations. The quantitative data showed that school staff increased 
their use of data, particularly relative to students’ performance in college classes; however, the 
difference between the treatment and comparison schools was not statistically significant. 


Table IV-8 shows the percentage of staff who reported engaging in various targeted continuous 
improvement activities at least once per month on the survey. 
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Table IV-8. 


Continuous Improvement Activities 


First Survey (2015) Fourth Survey (2018) Change 
Once a Once a 
Month Month Adj. Imp. 

Item Respondent | Level n Mean n + (%) Mean Mean Est. 
Our school All Program All Schools 
collected or Teachers 
received 
data on STEM 
student Program 
performanc Teachers 
e in college Program 
classes. Admin. 

Preyeive, || Regen | | it N/A N/A 

Comparison Comparison N/A 

Admin. 
Staff used All Program | AllSchools | 315 | 48.6. || 2.53. | 292 | 57.9 | 2.71 | 9.3 | 0.18" 
data on Teachers | CT | 284 | 50.4 | 256 | 87 | 62.20) 2.94 | 17 | 0.388 
student MI 184 47.3 2.51 205 56.1 2.62 8.8 0.11 
performanc STEM All Schools 136 52.2 2.57 135 55.6 2.67 3.4 0.11 
ein high Program “aT 55 | 45.5 | 2.44 | 39 | 436 | 259 | 19 | o15 |  #4| | 
school Teachers [Mi | | 81 | 568 | 2.65 | 96 | 604 | 271 | | 36 | 005 | #4| | 
classes to Program 
improve Admin. 
curriculum 
ane F Program vs. Prog. 12 3.02 0.20 
instruction. ce  frenaagerseesteeeeeecsaaneccececeedaneeeceeeneeeeeafecsaaaseeceeeeeeeeeecafcessaanneeeeeeeeeeceeeecaaauereesheceeeceeeeeeesaanneedbeceeeescssaanneeeeebeceeeeeeeecaaaaeeeeeeebeceeeeeeeesssauneceesbeeeecesteesaauaaeed 

Comparison Comp. 18 2.82 

Admin. 
lam All Program All Schools 315 47.7 2.59 292 49.3 2.60 1.6 0.01 
provided Teachers 
scheduled 
time to STEM 
work with Program 
other Teachers 
teachers on 
planning. Program 

Admin. 


Program vs. 
Comparison 
Admin. 


Note. * p< .05; ** p< .01; t p< .10. Response options ranged from “never” (1) to “once or twice a week” (4). 


For all teachers and administrators, results showed positive changes between 2015 and 2018 in 


schools’ collection/receipt of data on student performance in college classes. Twenty five 


percent (25%) of teachers and 38% of administrators reported collecting or receiving these data 


at least once a month in 2018. There were also positive changes over time in administrators’ 


and Bridgeport teachers’ reports on staff use of high school student performance data to 


improve curriculum and instruction. This positive change was not apparent among the STEM 


and Michigan teachers surveyed, possibly due to initial relatively high values on this scale. Fifty- 


eight percent (58%) of teachers and 60% of administrators reported using high school 


performance data to improve curriculum and instruction at least once a month in 2018. 
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Further, there were positive changes (though not statistically significant) over time in 
administrators’ reports on the extent to which they were provided scheduled time to work with 
teachers on planning. For teachers, changes were either negative or not apparent. Overall, 49% 
of teachers and 67% of administrators reported that they/staff were provided scheduled time 
to work with other teachers on planning at least once a month in 2018. SECEP administrators 
reported more frequent (though not statistically significant) use of data and more time to work 
with teachers on planning than did comparison administrators. 


Data from the interviews and site visits also showed that there was an increase in data usage 
across the two states. Unlike the surveys, however, the individuals interviewed reported 
increases in collaboration between teachers. As described in the Implementation Supports 
section, NCREST provided support around data usage. A discussion of how schools used that 
data occurs first and then the results are discussed separately by state. 


School Use of NCREST Data 


Throughout the grant, NCREST modeled data usage to administrators in both states. They 
shared school-level results from the student survey data they collected and results from the 
teacher survey collected as part of the evaluation. NCREST staff conducted workshops in how to 
interpret and use the data to make decisions. 


By Year 3 of implementation, all four Michigan ISD coordinators commented that they used 
NCREST student survey data and the teachers really appreciated it. For example, two leaders 
noted that they were concerned the surveys showed a loss in students’ interest in mathematics 
between Grades 8 and 12 and were trying to figure out how to address that concern. As one of 
them said, 


The [school staff] love the survey, and they feel like they're getting some answers to 
their questions. | know at one PD, they said that for the next few months, they were 
going to focus on one particular area, on the data that was collected, to find out about 
student disposition in math class, and what they could do to increase student, or 
improve student disposition, and finding where it's dropping off. 


By the end of the project, one district representative reported how the student survey results 
prompted them to begin collecting additional data not previously collected by the district to 
inform decisions around dual enrollment. 


Bridgeport 


Using Data. |n the early stages of implementation, district staff indicated a need to 
improve use of data from benchmark and interim assessments. By Year 2, Bridgeport had a 
district data team led by the district director of secondary education, who was also leading the 
SECEP project, and, according to district staff, an excellent data management department, 
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which provided data and reports for the team. The team met monthly and included guidance 
counselors and administrators from each high school and the SECEP project coordinator. The 
team looked at (1) testing data, such as SAT, PSAT, district standardized tests and classroom 
grades; (2) how many students were enrolled in AP and dual enrollment courses; (3) which 
courses they took; and (4) who were their teachers. This approach was continued in Year 3. 


Data were also utilized by the JFF leadership coach. A district leader noted that the leadership 
coach always based her discussion with principals on data, and curriculum coordinators also 
discussed data as a basis for instructional decisions at all their faculty meetings, 


The environment in the classroom, the environment in the hallway, everything is boiled 
down to data. And especially when they get their data back from the state. [The coach] 
has them scrub that data. 


While eight of 15 respondents stated that their school or district focused more on data in the 
past two to three years, most were not able to attribute the increase in quality and quantity of 
data use specifically to SECEP. Seven respondents reported that data had been a focus of the 
schools for several years. Nevertheless, some district and school staff highlighted differences 
that had occurred because of the project. One interviewed leader discussed how, as a result of 
the project, the district is drilling down more into specific data questions such as the 
relationship between taking certain dual enrollment classes and attendance or how students in 
certain classes or with certain teachers are performing academically. 


A principal at one high school stated how the school had been “stuck” prior to the grant in 
terms of leveraging data to make changes. This principal went on to say that the school is 
collecting and examining more data than in previous years, but more importantly, data are 
being explored at multiple levels (e.g., grade, department, different student groups) and that 
this deeper dive into the data has led to improvements in the way the staff identify areas for 
improvement. 


Collaboration and Common Planning. \|n terms of internal professional development 
and collaboration efforts, representatives from the middle and high schools that we visited 
during Year 4 indicated that collaborations have increased in recent years through weekly and 
monthly common planning times for teachers (which regularly involved data discussions). The 
interviews with school and district representatives indicated that, although all teachers had 
regular opportunities to collaborate through common planning time or other regularly 
scheduled meetings, the middle schools appeared to have more organized and structured 
activities around data monitoring, including data teams devoted to vertical (within grade 
and/or department) as well as horizontal data alignment (across grade). A high school principal 
reported how teacher collaboration was key to sustaining the work by saying, 
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| don't want success to be a hit one day and that we're at the point of slipping off 
again. | want it to have sustainability, whatever we do. Through common 
planning, we have the opportunity to do work through CIF, with the faculty, in 
which they work together using best practices, current research, and student 
data to make decisions on where the school is going to go. 


The principals at both high schools reported that teachers had taken more of a leadership role 
in organizing and facilitating collaborative efforts. 


Michigan 


Using Data. |n Michigan, data-use was a focus across all four years and each ISD was 
involved in data discussions. ISDs used a variety of other data to determine the readiness of 
their students for college courses. For example, at one district, a team looked at COMPASS 
data, algebra course pass rates, and credit audits. These data helped them to determine the 
state of students’ academic skills and prompted conversations about addressing the gaps 
through, for example, summer bridge programs. In another district, their use of data around 
students’ college performance revealed that students were lacking what they called “soft 
college readiness skills” and the district sought to develop strategies to address that concern. 


Although data-driven decision-making was a priority among all districts, most respondents that 
we interviewed did not attribute the focus on data to any particular program, including SECEP. 
Various initiatives contributed to the more intensive use of data, such as Michigan’s recent 
teacher evaluation system, the implementation of benchmark assessments across all four ISDs, 
and student tracking systems being implemented in three ISDs. However, four respondents that 
we interviewed did attribute the greater focus on data to SECEP. For example, one principal 
said that SECEP complemented the district’s efforts around data and that exposure to SECEP 
activities gave district leaders ideas about how to use data more effectively. 


Collaboration and Common Planning. Most respondents that we interviewed across the 
four ISDs reported that teachers were collaborating more regularly, with a specific focus on 
data. In three of the four ISDs, our Year 4 interviews suggest that teachers were given some 
form of individual or common planning time or professional learning community time to 
collaborate. Six respondents reported that collaboration had increased out of necessity given all 
the changes that staff were asked to implement, as one teacher put it, 


Increased collaboration, we've talked about that. This is something that has to 
happen with what we're doing. In order to make sure that our curriculum is 
college-aligned or students are successful in classes, there's even scheduling is a 
big issue with a middle college, just making sure that our schedule matches up. 
Because of those things, we've had to increase our collaboration. 
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Three respondents also indicated that, as part of this process, teachers were taking more of a 
leadership role in facilitating these collaborations. As one coach said, 


We have a lot of teacher-led professional development or teacher-led staff 
meetings or teacher-led training because certain teachers have taken on 
leadership roles in SECEP and are experts in certain areas. So, | think then, that 
has caused communication between administration and teachers to improve 
because you have more teachers that are taking on leadership positions. 


Program-Wide Summary 


Three themes emerged in both Bridgeport and Michigan with regard to the Culture of 
Continuous Improvement Design Principle: (1) both states reported using multiple data sources, 
including standardized assessments, college placement assessments, dual enrollment data, 
grades, and attendance and discipline records; three ISDs also reported using student surveys 
provided by NCREST; (2) while all schools were using data to some extent before SECEP, during 
the project, schools and districts increased their focus on data, analyzing it at a deeper level and 
more systematically; and (3) contrary to survey results, many interviewees reported that 
teacher collaboration had increased and teachers had taken on more leadership roles. 


Summary of SECEP Impacts on Design Principles 


As described in the previous sections, SECEP has changed the way in which schools were 
operating. The most substantial changes appeared to be in areas related to college-going, 
college readiness, college coursetaking, supports related to college readiness, and data usage. 
In the final administration of the staff survey, we asked respondents to consider whether 
changes in the school could be attributed specifically to SECEP. The survey asked nine questions 
about different potential impacts with response options ranging from “strongly disagree” (1) to 
“strongly agree” (4). It should be noted that these questions were considered cumulative, so we 
do not present any time comparison. As Table IV-9 shows, a majority of all respondents 
“agreed” or “strongly agreed” with all of the impact statements on the survey. 


Table IV-9. Perceptions of Implementation of the Early College Design Elements 
Fourth Survey (2018) 


Agree + 

Strongly 
Item Respondent Level n Agree (%) Mean 
Because of SECEP, our school is better | All Program Teachers All Schools 280 81.1 2.96 


able to prepare students for college. 


SENAY ee eamyeeeene te. | Oe acu oe re Beek 
| eee ane See eee a See) 
MI 91 84.6 3.04 
Program Admin. All Schools 42 88.1 3.12 
cT 16 87.5 3.19 
M | 2 | 85 | 3.08 
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Fourth Survey (2018) 


Agree + 

Strongly 
Item Respondent Level n Agree (%) Mean 
All Program Teachers All Schools 281 80.4 2.96 


Because of SECEP, our school has 
developed a culture of college 
awareness and readiness. 


STEM Program Teachers 


Program Admin. 


Because of SECEP, we have improved 
the instruction in our school. 


All Program Teachers 


STEM Program Teachers 


Program Admin. 


Because of SECEP, | have improved 
my own instruction. 


All Program Teachers 


STEM Program Teachers 


Program Admin. 


Because of SECEP, we implement 
more STEM projects. 


All Program Teachers 


STEM Program Teachers 


Program Admin. 


Because of SECEP, our school has 
increased focus on STEM subjects and 
careers. 


Because of SECEP, we have improved 
the supports that are in place for our 
students. 


All Program Teachers 


STEM Program Teachers 


_All Schools 


Program Admin. 


All Program Teachers 


_All Schools 


“MI 
All Schools 


STEM Program Teachers 


Program Admin. 


Because of SECEP, we have a stronger 
partnership with a postsecondary 
institution. 


All Program Teachers 


STEM Program Teachers 


Fourth Survey (2018) 


Agree + 

Strongly 
Item Respondent Level n Agree (%) Mean 
MI 79.3 3.01 
Program Admin. All Schools - 85.7 3.07 


All Program Teachers All Schools 277 69.7 2.79 
Because of SECEP, we better use i 
student performance data to improve MI h 195 68.7 2.79 
instruction and student outcomes. STEM Program Teachers All Schools 7 126 65.9 2.74 


Program Admin. 


MI 26 80.8 2.92 
Note. Response options ranged from “strongly disagree” (1) to “strongly agree” (4). It should be noted that these questions are considered 


cumulative, so we do not present any time comparison 


The highest proportion (over 80%) of teachers and administrators agreed that the project had 
resulted in an increased focus on college readiness. Administrators “agreed” or “strongly 
agreed” with the following impacts of the SECEP project: 


e Weimplement more STEM projects (97.6%), 
e Our school has increased focus on STEM subjects and careers (92.7%), and 
e Our school has developed a culture of college awareness and readiness (90.5%). 


Teachers most strongly agreed with the following impacts of the SECEP project: 


e Our school is better able to prepare students for college (81.1%), 

e Our school has developed a culture of college awareness and readiness (80.4%), 
e Our school has increased focus on STEM subjects and careers (77.9%), and 

e We have improved the instruction in our school (75.6%). 


The next sections of the report examine whether the school-level changes resulted in positive 
impacts for the students. 
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SECTION V: STUDENT IMPACTS—MICHIGAN 


The implementation supports and Design Principles described previously were intended to lead 
to improvement on a set of core outcomes, including an increased percentage of students 
earning college credit and reduced dropout rates. In this section, we present results from the 
quasi-experimental analyses of the impact of SECEP in Michigan. We also summarize perceived 
impacts on students and staff. 


Key findings concerning the impact of the project include the following. 


There was an eight percentage-point impact on the percentage of students enrolled in 
college-level courses, a difference that was not statistically significant. 

There was a Statistically significant 12 percentage-point impact on the percentage of 
students who received at least one college credit. 

Treatment students earned approximately twice as many college credits as comparison 
students, a difference that was statistically significant. 

There were positive impacts on college coursetaking and credits earned for 
economically disadvantaged students. 

There was a positive impact relative to college coursetaking and credits earned for non- 
minority students and no significant impact on underrepresented minority students. 
This has led to a widening gap between the two populations. Additionally, minority 
students participated in college courses at a rate less than one-third that of non- 
minority students. 

Impacts related to college coursetaking increased each year as the program became 
more mature, with Year 4 impacts higher than Year 3 impacts which were higher than 
Year 2 impacts. 

There was no statistically significant impact on dropout rates. 

A majority of school staff “agreed” or “strongly agreed” that SECEP had resulted in a 
range of positive effects on their school culture and students. 

In interviews, the most frequently mentioned project impacts were related to an 
increased orientation toward post-high school plans for students, expanded access to 
dual enrollment courses, and increased collaboration among teachers. 


Methodology 


The evaluation used a quasi-experimental design to assess the impact of the project on 
outcomes in two domains: (1) college credit-bearing courses and (2) dropping out of school. 
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Sample 


The study sample in Michigan included 11 treatment high schools? and 42 comparison schools 
that were similar to the 11 treatment schools on baseline measures of the outcomes and key 
demographic characteristics. We began by identifying a potential pool of comparison schools. 
Characteristics of the treatment schools were used to establish inclusion criteria for potential 
control schools. Because all treatment schools were classified as general education schools, 
specialized schools were excluded (e.g., alternative, special education, vocational/CTE, magnet, 
IB, and charter schools). In addition, because some treatment schools were combined middle 
and high schools, the list of potential comparison schools also included schools that served 
middle and high school students. In our original matching we identified 44 comparison schools 
(four schools for each treatment school). However, after conducting the matching, we learned 
that one of the comparison schools had an Early College operating within the school, and one 
school closed during the study period. Both schools were excluded from the study. 


We employed a matching procedure entitled “nearest neighbor match” with Mahalanobis 
distance criteria, using the R software program MatchlT version 3.02 (Ho et al., 2011). The 
outcome measures and characteristics on which the schools were matched included the 
following indicators all measured in the baseline year (2013-14): 


e School type. Because several schools in the treatment group were combined middle 
school and high school, we ensured that schools were matched within type (i.e., high 
school only, middle and high school). This information was available at the school level. 

e Percentage of students dropping out of school. We used the four-year cohort dropout 
rate calculated at the school level by the Michigan Department of Education. 

e Percentage of students receiving some college credit. This was the percentage of 11‘ 
and 12" graders who had taken and received college credit in the 2013-14 school year 
through one or more potentially college credit-bearing courses, including AP and dual- 
enrollment courses. This variable was calculated from student-level data provided by 
the Michigan Department of Education and rolled up to the school level. 

e Average number of college credits earned. This measure looked at the average number 
of college credits earned by 11*- and 12*-grade students in the 2013-14 school year 
either by passing the AP exam or by passing college credit classes (see outcomes section 


3 At the end of 2017, one of the treatment schools closed. The majority of students who attended this school 
enrolled in two treatment schools within the same district in 2018. For purpose of assigning baseline school-level 
covariates, students who transferred out of the closed treatment school at the end of 2017 were assigned the 
baseline school-level covariates for the school they were enrolled in in 2018. The final outcome analysis included 
11 treatment schools in 2017 and 10 treatment schools in 2018. 
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below for a more detailed description on this variable). Similar to the previous variable, 
this variable was calculated from student-level data provided by the Michigan 
Department of Education and rolled up to the school level. 

e Percentage of students who were identified as economically disadvantaged. These 
data were available at the school level. 

e Percentage of students who identified as African-American, American Indian, or 
Hispanic/Latino. These data were available at the school level.* 

e School enrollment. This was the number of students enrolled in the entire school. For 
schools that contained both middle and high school students, the school size variable 
reflected enrollment for both grade configurations. 

e National standardized test score. This was the average ACT composite score (math, 
reading, English, and science) for students in the school, converted to z-scores. 

e State proficiency. This was the average scale score on the math and reading portions of 
the Michigan Merit exam, also converted to z-scores. 


In order for the treatment and comparison samples to be considered equivalent by the What 
Works Clearinghouse (WWC) standards, the differences between the two groups must be less 
than .25 standard deviations. We assessed baseline equivalence on baseline measures of the 
outcomes and on key demographic characteristics. As Table V-1 shows, the difference between 
the two groups was less than .25 standard deviations on all outcome measures and 
demographic characteristics. It is important to note that we took the most conservative 
approach to assessing baseline equivalence, calculating the differences at the school level. No 
data used to calculate baseline equivalence were imputed at either the school or student level. 


Table V-1. Baseline Equivalence, Treatment and Comparison Groups, Michigan 


Treatment Comparison 
Mean Mean Standard Effect 
(SD) (SD) Deviation Size 
Measure N=11 N=42 (Pooled) (Hedge’s g) 
Cohort dropout rate 3.657 4.032 2.965 124 
(2.074) (3.144) 
% of 11'- and 12**-grade students receiving at least one 6.958 6.268 8.437 .081 
college credit (9.991) (8.013) 
Average college credits earned 518 453 .796 .081 
(.783) (.799) 
% Economically disadvantaged 47.013 47.044 17.467 .002 
(14.830) (18.052) 
% Minority 28.852 24.911 31.813 .153 
(31.890) (31.795) 
School enrollment 622.091 637.929 425.359 .037 
(367.403) (438.333) 


4 For schools that were combined middle and high schools, we calculated the percentage of economically 
disadvantaged and racial minority students in the high school from the student-level data. 
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Treatment Comparison 
Mean Mean Standard Effect 
(SD) (SD) Deviation Size 
Measure N=11 N=42 (Pooled) (Hedge’s g) 
ACT (z-score) 18.482 18.629 1.756 .082 
(1.910) (1.717) 
Michigan Merit Exam Scores for Reading and Math (z-score) 1098.333 1099.740 10.563 131 
(11.133) (10.419) 


We also calculated baseline equivalence for the two sub-groups of interest: (1) economically 
disadvantaged students and (2) students who were members of racial or ethnic groups 
underrepresented in college (American Indian, African-American, and Hispanic/Latino). Baseline 
equivalence for the sub-groups is provided in Appendix D. 


A note about sample representativeness: No students were dropped from the baseline 
equivalence calculations because of missing data; as a result, 100% of enrolled students were 
included in both the dropout and the college coursetaking outcomes. The outcome analytic 
samples included only students who had complete baseline covariates (achievement data and 
demographic characteristics). As such, not all students were included in the sample. For the 
dropout sample, 85.1% of the enrolled treatment group students in the relevant grades 
contributed data to the outcome analysis, and 86.3% of enrolled comparison group students in 
the relevant grades contributed data to the outcome analysis. For the college course taking 
analyses, 85.1% of the enrolled treatment group students in the relevant grades contributed 
data to the outcome analysis, and 85.7% of enrolled comparison group students in the relevant 
grades contributed data to the outcome analysis. 


Outcomes 


All outcomes used student-level data provided by the Michigan Department of Education. Thus, 
the first two outcomes were binary and the third was interval at the student level and discussed 
below as percentages and averages, respectively. It is important to note that no outcome 
measures were imputed. 


e Percentage of students enrolled in at least one college credit-bearing course. This 
measure examined the percentage of 11*- and 12*-grade students in the school who 
enrolled in at least one college bearing course at any point in their high school career, 
identified in the data as either dual enrollment or AP. These results were examined for 
the 2015-16, 2016-17, and 2017-18 years. 

e Percentage of students who received at least one college credit. This measure 
examined the percentage of 11'*- and 12'*-grade students in the school who enrolled in 
at least one college-level course and received college credit for that course at any point 
in their high school experience. Students could have received credit either through 
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enrolling in at least one AP course and passing the AP exam or enrolling in at least one 
dual enrollment course in any year of high school and passing the course. These results 
were examined for the 2015-16, 2016-17, and 2017-18 years. 

e Number of potential college credits received. This outcome was the average number of 
potential college credits received by 11'- and 12"-grade students. We use the term 
potential college credit because only postsecondary institutions can actually award the 
credit. Potential college credits could be earned in two ways. First, we obtained scores 
for students who took AP exams. Students were awarded potential college credit using 
AP transfer equivalencies from the University of Michigan-Flint campus. The second way 
for a student to earn potential college credit was through receiving credit for a dual 
enrollment course. For students who passed a dual enrollment course (i.e., indicated as 
a “pass” in the transcript data), in order to obtain an estimate of the total number of 
credits earned, we multiplied the number of classes completed by three credits. This 
outcome was cumulative and included credits that students in the sample had earned in 
previous years and in the current year. These results were examined for the 2015-16, 
2016-17, and 2017-18 years. 

e Dropping out of school. This outcome was the proportion of students in the high school, 
Grades 9-12, who were identified as dropping out in the 2016-17 and 2017-18 school 
years. 


Analysis 


The difference between SECEP and comparison schools was examined using hierarchical linear 
modeling (HLM) implemented with SPSS (version 25) Linear Mixed Model, which accounts for 
the fact that students are nested within schools (SPSS, 2005). In addition to including a binary 
indicator for treatment status, the models also included grade and study year as design factors 
and the following variables were included in the models as covariates: 


e Student level. Middle school math and reading scores (standardized as z-scores), 
minority status, economically disadvantaged status, English Language Learner status, 
exceptionality status, and gender. 

e School level. Baseline measure of the percent of students receiving at least one college 
credit, baseline percent minority, baseline percent economically disadvantaged, school 
enrollment, baseline school dropout rate, and baseline academic performance. 


Because the three college course outcomes were cumulative and because we were pooling 
data in our sample across three years, this posed a special challenge for our analysis. It was 
possible for students to appear twice in the analytic data file (for example, students who 
enrolled in high school in 2014-15 could appear once as an 11'*-grade student in 2016-17 and 
again as a 12'"-grade student in in 2017-18). For these cases, the 12'"-grade record was retained 
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for analysis because it would have captured accumulated college credit through 12" grade. If a 
student was present in 11° grade in 2016-17, but not present in 12* grade in 2017-18, the 11*- 
grade record was retained for analysis. 


We repeated the above analyses for the two sub-groups of interest, economically 
disadvantaged students and minority students. 


Impact on Enrollment in College Credit-Bearing Courses 


As described above, we explored two different outcomes related to enrollment in college 

credit-bearing courses: (1) enrollment in at least one college credit-bearing course and (2) 
receipt of at least one college credit. We first begin by looking at enrollment in the college 
courses. 


As shown in Table V-2, there was an overall impact on college-level course enrollment of 7.7 
percentage points. This difference was not statistically significant at the standard value of 
ps.05. However, when looking at the impacts by type of college-level courses, we see a 
Statistically significant impact of 13.0 percentage points on enrollment in dual enrollment, 
which is partially offset by a nine percentage-point drop in enrollment in AP courses. 


Table V-2. Impacts on the Percentage of 11" and 12" graders Enrolled in at least One College 
Credit-Bearing Course, Michigan 


Treatment Comparison Adjusted Impact 
Impact Estimate 
Estimate at Standard 


Adjusted 

Outcome Sample Mean (SD) Sample 

% 11" and 12* graders enrolled in 5,230 45.23 20,863 
any college course (49.44) 

% 11" and 12" graders enrolled in 5,230 20.48 20,863 
dual enrollment courses only (35.98) 


% 11° and 12" graders enrolled in 5,230 16.98 20,863 
AP courses only (37.88) 

% 11" and 12" graders enrolled in 5,230 7.89 20,863 
both AP and dual enrollment courses (29.82) 

% enrolled in any college course-- 1,527 34.20 6,694 
underrepresented minority students (43.40) 

% enrolled in any college course— 3,703 45.95 14,169 
NON-minority students (50.01) 

% enrolled in any college course— 2,462 35.51 9,478 
economically disadvantaged (46.17) 

students 

% enrolled in any college course— 2,768 51.99 11,385 
NON-economically disadvantaged (49.92) 

students 

% enrolled in any college course— 15,272 
11"-grade students (45.27) 


% enrolled in any college course— 3,706 48.98 14,815 
12'-grade students (49.90) 


41.88 
(49.34) 
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Results related to earning at least one college credit were also analyzed for specific sub-groups 


of students: (1) underrepresented minority students, (2) students who were not 


underrepresented minority (White or Asian), (3) students who were economically 


disadvantaged, and (4) those who were not economically disadvantaged. The results were also 


analyzed by grade. None of the sub-groups examined had statistically significant impacts. In 


particular, there was an impact of close to zero on the enrollment of minority students in any 


college credit-bearing courses. When we broke the minority enrollment out by AP and dual 


enrollment courses, we see that minority students are much more likely to be enrolled in AP 


courses than in dual enrollment courses; in fact, a higher percentage of minority students in 
SECEP schools were enrolled in AP (19.6%) than non-minority students (16.4%). The reverse 
was the case with dual enrollment, where 12.1% of SECEP minority students were enrolled only 


in dual enrollment course compared to 20.9% of SECEP non-minority students. These detailed 


sub-group analyses are provided in Appendix E. 


The next outcome examined was the percentage of 11'-and 12*-grade students who had 


earned at least one college credit. As Table V-3 shows, there was a statistically significant 12 


percentage-point impact on the percentage of students receiving at least one college credit. 


Table V-3. Impacts on the Percentage of 11th and 12th Graders Receiving at Least One Potential 


College Credit, Michigan 


SERVE 


Treatment Comparison Adjusted Impact 
Impact Estimate 
Adjusted Unadjusted Estimate Standard 
Outcome Sample Mean (SD) Sample Mean (SD) at Year 4 Error p-value 
% 11"- and 12**-grade students who 5,230 29.76 20,863 17.48 12.28 3.30 <.001 
earned at least one college credit (44.16) (37.98) 
% 11*- and 12''-grade students who 5,230 22.86 20,863 9.97 12.90 3.87 <.01 
received credit only through dual (40.99) (29.95) 
enrollment 
% 11*- and 12''-grade students who 5,230 4.57 20,863 6.60 -2.03 1.69 .24 
received credit only through AP (16.25) (24.82) 
courses 
% 11*- and 12''-grade students who 5,230 2.37 20,863 92 1.45 .64 .03 
received credit through both AP and (15.51) (9.52) 
E dual 
% Underrepresented minority 1527 14.68 6694 8.90 5.78 3.00 .06 
students who earned at least one (33.96) (28.48) 
college credit 
% NON-minority students who 3703 32.29 14,169 21.53 10.77 3.69 .01 
earned at least one college credit (46.65) (41.10) 
% Economically disadvantaged 2,462 20.69 9,478 10.17 10.52 2.95 <.01 
students who earned at least one (37.62) (30.23) 
college credit 
% NON-Economically disadvantaged 2,768 36.17 11,385 23.56 12.61 3.77 <.01 
students who earned at least one (47.70) (42.44) 
college credit 
% Grade 11 students who earned at 3,866 21.04 15,272 11.42 9.62 3.07 <.01 
least one college credit (39.51) (31.81) 
% Grade 12 students who earned at 3,706 31.57 14,815 20.26 11.32 3.64 <.01 
least one college credit (44.76) (40.19) 
54 


Note. The adjusted treatment mean is calculated by taking the unadjusted comparison mean and adding the adjusted impact estimate. 


Why was there a statistically significant impact on earning college credit but not on enrollment 
in college-level courses? As shown in the table, we see that, overall, more students received 
college credits through dual enrollment than through AP, despite an overall higher percentage 
of students enrolled in AP courses (see Table V-2). We conducted follow up descriptive analyses 
that showed that students were much more likely to earn college credit when they took dual 
enrollment courses than when they took AP courses. The percentage of dual enrollment 
courses that resulted in potential college credit was over 90%. On the other hand, the 
percentage of AP courses that resulted in potential college credit (earned by taking the exam 
and receiving a passing score) was less than 10% in our sample. 


When looking at impacts by sub-group, we see a 6 percentage-point impact on minority 
students’ completion of at least one college credit; a result that was statistically significant at 
p=.06. This was compared to a statistically significant impact of 10.8 percentage points on non- 
minority students. In looking within the treatment group only, 14.7% of minority students 
earned at least one college credit compared to 32.3% of non-minority students. 


The impact on economically disadvantaged students was a statistically significant 10.5 
percentage points compared to a significant 12.6 percentage-point impact on non-economically 
disadvantaged students. When examined by grade, the impact was larger for 12*-grade 
students than for 11*-grade students. More detail on the sub-group analyses can be found in 
Appendix E. 


Impact on College Credit Attainment 


The third outcome related to college coursetaking was the number of potential college credits 
earned. This outcome incorporated any increase in access to college courses as shown under 
the previous outcome, coupled with any increase in the number of courses students were 
taking. 


Table V-4 shows the total number of credits earned by 11" and 12* graders. As the results 
show, the number of credits earned by treatment students was approximately double the 
number of credits earned by comparison students (i.e., 3.5 credits for treatment students 
compared to 1.8 credits for comparison students). The table also shows that the majority of 
these credits were earned via dual enrollment courses, with treatment students earning 
approximately 2.5 times as many dual enrollments credits as comparison students. Regarding 
AP credits, treatment students earned slightly (although non-significantly) fewer credits from 
AP courses than comparison students did. 
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Table V-4. Impacts on 11'- and 12‘-Grade Students’ Number of Potential College Credits 


Earned, Michigan 


Treatment Comparison Adjusted Impact 
Impact Estimate 
Sample Adjusted Unadjusted Estimate at Standard 

Outcome Mean (SD) Sample Mean (SD) Year 4 Error p-value 
Average # of credits earned 5,230 3.53 20,863 1.75 1.78 42 <.001 

(8.10) (5.49) 
Average # of credits earned through 5,230 2.94 20,863 1.10 1.84 48 <.001 
dual enrollment (7.86) (4.43) 
Average # of credits earned through AP 5,230 59 20,863 65 -.06 19 76 
courses (1.60) (3.30) 
Average # of credits earned through 5,230 41 20,863 14 .27 08 <.01 
both AP and dual enrollment (2.83) (1.74) 
Average # of credits earned by 1,527 1.31 6,694 78 53 47 26 
underrepresented minority students (4.41) (3.75) 
Average # of credits earned by NON- 3,703 4.08 14,169 2.21 1.86 58 <.01 
minority students (9.09) (6.09) 
Average # of credits earned by 2,462 2.44 9,478 0.95 1.49 39 <.001 
economically disadvantaged students (6.17) (3.99) 
Average # of credits earned by NON- 2,768 4.61 11,385 2.42 2.18 72 <.01 
economically disadvantaged students (9.33) (6.40) 
Average # of credits earned by Grade 3,866 1.77 15,272 89 0.88 .29 <.01 
11 students (4.12) (3.26) 
Average # of credits earned by Grade 3,706 4.43 14,815 2.13 2.03 69 < .001 
12 students (9.21) (6.16) 


When examining the impact by sub-groups, we saw no significant impact on credits earned by 


underrepresented minority students and a significant, positive impact on credits earned for 


non-underrepresented minority students. In the treatment group, the number of credits earned 


by minority students was approximately one-third the amount earned by non-minority 


students; while in the comparison group, minority students earned slightly less than one half 


the number of credits as non-minority students. Both economically and non-economically 


disadvantaged students saw significant and positive impacts, with a larger impact seen by non- 


economically disadvantaged students. The impact on 12" graders was larger than the impact 


on 11" graders. 


Changes Over Time 


The impacts reported above combine three cohorts of students, from the second, third, and 


fourth years of implementation. Because the STEM Early College Model is complex and requires 


the establishment of systems over time, we might expect to see impacts increase as the 


program becomes more mature. In other words, we might expect examples for students in the 


fourth year of implementation (2017-18) to be higher than impacts were for the first cohort we 


looked at (2015-16). As a result, we explored impacts by year. As Table V-5 shows, we do see 


larger impacts as the program matures. For example, the impact on enrollment in college 
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credit-bearing courses was only 3 percentage points in 2015-16 but it was 16.5 percentage 
points and statistically significant in 2017-18. 


Table V-5. Impact on College Credit-Bearing Courses, by Year of Implementation, Michigan 


Treatment Comparison Impact 
Estimate 
Sample Adjusted Unadjusted Impact Standard p- 

Outcome Year Mean (SD) Sample Mean (SD) Estimate Error value 

% 11" and 12* graders 2015-16 2,600 36.36 10,243 33.37 2.99 4.75 0.53 
enrolled in any college (45.82) (47.16) 

course 2016-17 2,539 43.57 10,198 35.37 8.20 4.27 0.06 
(48.82) (47.81) 

2017-18 2,535 53.15 9,968 36.62 16.53 4.93 <.01 
(49.76) (48.18) 

% 11'*- and 12'*-grade 2015-16 2,600 20.91 10,243 13.22 7.70 3.44 0.03 
students who earned at (38.87) (33.87) 

least one college credit 2016-17 2,539 25.59 10,198 16.53 9.06 3.23 <.01 
(42.07) (37.15) 

2017-18 2,535 33.42 9,968 17.48 15.95 3.93 <.001 
(46.16) (37.98) 

Average # of credits 2015-16 2,600 2.02 10,243 1.15 0.87 0.32 <.01 
earned (5.04) (4.18) 

2016-17 2,539 3.05 10,198 1.57 1.48 0.47 <.01 
(8.03) (5.06) 

2017-18 2,535 4.38 9,968 1.80 2.58 0.55 <.001 
(9.23) (5.59) 


Exploring Sub-Group Gaps 


As described above, there were gaps in attainment between certain sub-groups. For example, 
in looking at the percentage of students who earned at least one college credit, we saw that 
14.7% of underrepresented minority students in SECEP schools earned at least one college 
credit. This was more than the 8.9% of underrepresented minority students earning at least one 
college credit in comparison schools, but it was also substantially less than the percentage of 
non-minority students earning at least one college credit (32.3% in SECEP schools). Table V-6 
below summarizes the sub-group findings. 


When we looked at the size of the difference—the gap between the percentage of minority 
students in a school taking college-level courses and the percentage of non-minority students 
taking college-level courses—we saw that the gap in earning at least one college credit 
between minority and non-minority students was larger in SECEP schools than in comparison 
schools. For example, the gap between the percentage of minority and non-minority students 
earning at least one college credit in SECEP schools was approximately 18 percentage points in 
SECEP schools compared to 13 percentage points in comparison schools. On the other hand, 
when looking at gaps between economically disadvantaged and not economically 
disadvantaged students, they were essentially the same in both SECEP and comparison schools 
(15 percentage points in SECEP schools and 13 percentage points in comparison schools). 
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To explore the gaps in more depth, we tested whether there were also gaps in enrollment in 
the courses (gaps in access) and gaps in successful completion once a student was enrolled in 
the course. As Table V-6 shows, there was also a gap in enrollment between underrepresented 
minority and non-minority students and between economically disadvantaged and not 
economically disadvantaged students. This suggests that underrepresented minority and low- 
income students enrolled in lower numbers in college-level courses in both treatment and 
comparison settings, with a larger enrollment gap for minority students in SECEP schools than 
in comparison schools. 


Table V-6. Summary of Sub-Group Gaps 


Underrepresented Minority Status Economically Disadvantaged Status 
SECEP Comparison SECEP Comparison 
Not Not 
Outcome Minority Minority Minority Minority ED Not ED ED Not ED 
% of 11 and 12 graders 34.2 i 46.0 33.6 i 39.4 35.5 i 52.0 28.1 i 45.3 
lled i II eens eee eee |e REESE! eA ee eee eee eee eee ene nen: Seer een eee | Meee eee een Keener etme mar 
On ae eee Gap=11.8 Gap=5.8 Gap=16.5 Gap=17.2 
course 
% of 11" and 12" graders 147. } 32.3 8.9 i 21.5 20.7 {| 362 10.2 | 23.6 
iui t least Tl ener etetene terete’ tet ter ttelene reetetctem ee eens tetera es Utrerer eleven tneem Meee eieletetetereerri tne! Urarttey er enero nce ate ne lettclenarteent! Ler teer eter ter aere em 
Cy eis eee nee eae Gap=17.6 Gap=12.6 Gap=15.5 Gap=13.4 
credit 
43.0 70.2 26.5 54.6 58.3 69.6 36.3 52.1 
% of students who earned at 
least one college credit When pee pret rar teeter etnentanetane ectaretaeenenaaneaeeaartacegaeraeraaraaeaaeaaepaaraaeaaraaeaaeorsacasertaertsareaarraaesaarad 
’ Gap=27.2 Gap=28.1 Gap=11.3 Gap=15.8 
taking college-level courses 


Note. The sample for the first two outcomes is the full population of the specific sub-group. The sample for the third outcome is only those 
students who took at least one college-level course. 


When we looked at successful completion of courses once students are enrolled, we saw that 
gaps were present in that situation as well. The table shows that once a minority student was 
enrolled in a college-level course, the gap in successful completion rates between minority and 
non-minority students was larger than the gap in enrollment, but it was similar in both SECEP 
and comparison schools (27 percentage points in SECEP schools and 28 percentage points in 
comparison schools). It is also important to note that all students in SECEP schools were more 
likely to receive potential college credit for their classes than were similar students in 
comparison schools. 


A possible mechanism for the increased success in SECEP college courses may be related to 
differences in AP and dual enrollment coursetaking. As described earlier, underrepresented 
minority students were more likely to take AP courses than dual enrollment courses, but AP 
courses were also less likely to result in potential college credit. Thus, part of the explanation 
appeared to be driven by the types of courses to which students were given access. 


These findings suggest that the gaps between minority and non-minority students and between 
economically disadvantaged and not economically disadvantaged students are due to inequities 
in both access, particularly to dual enrollment courses, and in successful completion. 
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Impact on Dropouts 


As part of the Early College Model theory of change, students are expected to gain more of a 
vision of their future and provide students with additional supports. These components of the 
model are expected to work together to keep more students in school. As a result, the 
evaluation examined the impact on the percentage of students in Grades 9-12 who dropped 
out of high school in either the third or fourth years of implementation (2016-17 and 2017-18 
school years). As Table V-7 shows, there were no statistically significant differences between 
treatment and comparison groups, although dropout rates were descriptively lower overall and 
for all sub-groups (with the exception of non-minority students) in the treatment schools. 


Table V-7. Impacts on Percentage of Students Dropping Out of School, Michigan 


Treatment Comparison Adjusted Impact 
Impact Estimate 

Adjusted Unadjusted Estimate Standard p- 
Outcome Sample Mean (SD) Sample Mean (SD) at Year 4 Error value 
% of students who dropped out of 10,528 1.63 41,092 1.84 -.21 46 .66 
school (11.33) (13.44) 
% Underrepresented minority students 3,015 2.41 13,069 2.96 -.55 76 48 
who dropped out of school (13.50) (16.94) 
% NON-minority students who 7,513 1,53 28,023 1.30 wae Al 59 
dropped out of school (10.28) (11.35) 
% Economically disadvantaged 5,216 2.46 19,564 2.66 -.19 .62 76 
students who dropped out of school (13.80) (16.08) 
% NON-Economically disadvantaged 5,312 .67 21,528 1.05 -.38 36 30 
students who dropped out of school (7.91) (10.21) 


Perceptions of Impacts of the Program on Staff and Students 


In addition to measuring impact using student administrative data, we also explored 
stakeholder perceptions of the impact of the program using multiple sources of data, including 
the staff survey and site visits. We begin by presenting the findings from the impact survey and 
then share how staff and students discussed the impact of the program. 


Survey Results 


On the staff survey, we asked participants to indicate the extent to which they agreed with a 
set of statements around the potential impacts of the program on school culture and students 
with response options ranging from “strongly disagree” (1) to “strongly agree” (4). Table V-8 
presents the percentage of Michigan respondents who “agreed” or “strongly agreed” with six 
statements. The results are broken out by role type (i.e., all program teachers, STEM program 
teachers, and administrators). 
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Table V-8. Perceptions of Impact on School Culture and Students— Michigan Respondents 


Item 


Respondent 


Fourth Survey (2018) 


Agree + Strongly 
Agree (%) 


Because of SECEP, the staff in our school 
have higher expectations for all students. 


Because of SECEP, we see positive changes 
in student engagement, academics, or 
interest in STEM areas. 


All Program Teachers 


Program Administrators 
All Program Teachers 


Program Administrators 


Because of SECEP, there is an increased 
collaboration among teachers. 


All Program Teachers 


Program Administrators 


Because of SECEP, trust and collaboration 
among all staff has improved. 


All Program Teachers 


Program Administrators 


Because of SECEP, our school is more 
focused on a common vision and mission. 


All Program Teachers 
STEM Program Teachers 
Program Administrators 


Because of SECEP, relationships between 
teachers and students have improved. 


All Program Teachers 


Program Administrators 


A majority of respondents “agreed” or “strongly agreed” that SECEP had resulted in impacts on 


school culture and on students. The highest perceived impact was in the area of increasing 


student interest and engagement. Over 70% of all teachers and 80% of administrators “agreed” 


or “strongly agreed” that SECEP had resulted in positive changes in student engagement, 


academics, or interest in STEM areas. The area with the lowest agreement in terms of perceived 


impact for administrators (69.2%) and teachers (55.8%) was improvement of trust and 


collaboration among all staff. 


Site Visit Results 


In addition to the survey, participants were asked to reflect on the impact of the program 


during interviews conducted as part of the site visits to eight schools across the four ISDs. These 


interviews expounded on many of the same themes seen in the survey. 


Impacts on School Culture 


In three of the schools visited, staff reported that the grant facilitated the development of a 


more cohesive vision for their school. One staff member noted, “So, | think as a building we've 


come together a lot more. As a family instead of being individuals in a classroom.” A colleague 


in the same school said, “I think this grant has helped in [unifying] the high school and.... Giving 


a common goal of how to step it up has provided ways, methods, ideas to help step it up.” 


A total of 21 staff in six schools described an increase in collaboration among teachers, 


primarily because of the need to coordinate and work together on projects being implemented 


by the school and the professional development opportunities provided by the grant. One staff 
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member noted, “...huge collaboration among teachers. They talk more. They're in the 
conferences more, so there's more communication about lessons, and ideas, and units.” 


Nine respondents from six schools reported that teacher leadership had increased because of 
the opportunities the grant gave teachers, either to lead in-house professional development or 
to present at state and/or national conferences. As one principal said, 


| think this project has done wonders to build capacity within the staff, because it has 
allowed me to bring numerous staff members to national conferences and present, 
which they never would have. All of a sudden, you present at a national conference, and 
you get people coming and asking you questions and people coming up and shaking 
your hand and saying thank you and emailing you, "Hey, can you help me?" That builds 
the staff confidence. And it builds their self-image. So now, all of a sudden, in their 
minds, they're a leader in the educational community in this, whether it's coaching or 
inquiry instruction or a place-based education, all of which we've done presentations at 
the national level on. 


A teacher in a different school described the impact this way: 


| went from being somebody who just followed directions and did what | felt was good 
enough to now, I'm looking for every opportunity to improve and do something 
interesting and new. I'm not worried about boxes, I'm not worried about putting in extra 
effort. I'm making things, which has inspired me to be better. | like this job more now 
because I'm doing things that | feel are valuable, rather than just following directions. 


For one school, one of the primary benefits of the project has been the way in which their 
school has been perceived by the community. Five interviewees associated with that school 
commented that their school was being viewed more positively and, as a result, the district was 
beginning to attract more students: 


And so, | think now, having this emphasis on college readiness and being able to throw 
out numbers, to have data to say, like, "These are all our students right now that are 
dual enrolled. And this is how they're doing in their classes." That has positively 
impacted the view of our school. 


On the other hand, faculty in a different school noted a general decline in school climate and 
culture because of the project, the Early College work, and the subsequent increase in students 
taking college courses. Three faculty at this school commented that the Early College was 
pulling higher achieving students out of the building, leaving lower-performing students behind. 
Two also noted that some high school positions were being lost and that fewer AP courses were 
being offered. 
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Student Impacts 


Respondents reported a variety of student impacts coming from increased use of project-based 
learning, increased access to advanced courses, and a more postsecondary-oriented culture. 


Fifteen interviewees from six schools described how the project has led to students paying 
increased attention to their postsecondary goals and have more of a vision for the future. One 
staff member said, 


| think just for so many of our kids in [our] schools, they thought that college was never 
an option for them, whether it's due to financial reasons or just low confidence. “My 
parents didn't go to college. I'm not smart enough to go to college.” | think we are 
starting to change that attitude in most of our students, well, many of our students. | 
think they're starting to see, once they get into college and they see that they're able to 
take classes, and they're passing the classes, and it's paid for, | think now they're 
starting to change their mindset. 


Fourteen teachers from five different schools reported that the use of projects led to increased 
student engagement and active learning in the classroom. A high school staff member 
described the type of instruction she saw in a classroom whereby the teacher was 
implementing strategies from professional development received as part of the grant: 


| go into a science classroom. This is a science room in which | see students of all 
different levels. I've got some that greatly struggle, some that are high achieving. 
They're sitting in groups, they are working on white boards, the teacher has posted a 
friction example on the board and the students are diagramming it out, they're labeling, 
they're designing, they're doing different things and the discussions I'm seeing are 
phenomenal. I’m seeing students work together who never would have worked 
together in the past. I'm seeing students who weren’t participating that are now holding 
quality discussions with their peers and volunteering and reporting out. 


High school students in one school noted that the use of more hands-on instruction had 
increased their interest in science. Six staff from two schools mentioned that the change in 
instruction was also leading to improved student behavior, with one school indicating that their 
number of suspensions had dramatically decreased. 


Across all the high schools we visited, participants commented on the increase in students 
taking dual enrollment or other advanced courses. Also, across all the high schools, faculty 
members reported that students received positive benefits from those courses. For example, 
one staff member said, “It's giving some students whose family socio-economic status probably 
would have prevented them from going to college to get the opportunity to go, and | think 
that's powerful for those families.” A college faculty member agreed that students were leaving 
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high school more prepared for college, attributed, at least in part, to the increased exposure to 


college courses: 


So, these students are coming to college, not only are they getting credits, they're more 
prepared for, whether it's moving on to the university or staying at [the] college, they're 
more prepared for that work. We're noticing now, [that] students have a much better 
grasp of the career pathway because of the work in alignment and the current 
exploration that's being done at the districts as well. 


One middle school reported that the expanded access to college courses in the high school had 


trickled down such that the middle school was offering more advanced courses: 


We're now teaching high school credits in the middle school. Computer, world language, 
and algebra | is now in the middle school, so those credits are being taken here, allowing 
more kids to take dual enrollment at the high school level, so some of their numbers 
that they're so proud of, which is an amazing increase, is attributed to this trickle-down 
through this grant that these kids are taking their credits early. 


Although many perceived impacts were positive, two faculty members in one high school, 


believed that having students in dual enrollment courses meant that they were paying less 


attention to their high school courses. One faculty member commented: 


I've [had] to remind [students,] when they're in my class, and they've said, "Well, I've 
got this college paper due." Well, that's not more important than my class.... No, you 
can't be typing on a laptop, you need to be participating in what we're doing because 
it's necessary for you to pass my class. 


In summary, the quantitative and qualitative data suggest that schools are making changes that 


are resulting in positive impacts in school culture and improving student outcomes. The next 


section presents results for Connecticut. 
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SECTION VI: STUDENT IMPACTS—CONNECTICUT 


This section presents the student impacts coming from the implementation of the project in 
Bridgeport, Connecticut. We present the research design used, the impact on core outcomes, 
and perceived impacts on students and staff. 


Key findings concerning the impact of the project include the following. 


e The impact study suffered from various design and data collection challenges that 
meant we could not make causal claims about the impact of SECEP on the schools. One 
of the key challenges was that the three treatment schools were among the most 
disadvantaged in Connecticut, which made it impossible to find comparison schools that 
were equivalent on both baseline measures of the outcomes and on demographic 
characteristics. Any findings need to be considered in light of these caveats. 

e Both treatment and comparison schools had higher percentages of students taking 
college-level and CTE courses in Year 4 than in the baseline year, however, some of this 
was due to a change in reporting by Connecticut as CTE was included in the measure in 
Year 4 but not in the baseline year. When adjusted for differences in poverty and race, 
the estimated levels of enrollment were similar in both treatment and comparison 
schools. 

e Dropout rates in the treatment schools increased over the life of the project; in 
comparison schools, dropout rates decreased. Differences between the two groups of 
schools were not statistically significant. 

e Both treatment and comparison schools increased their 9'*-grade enrollments in college 
preparatory courses. When adjusted for initial differences in poverty and race, the 
increases were higher in the treatment schools, but the difference between the two 
groups was not statistically significant. 

e Participants reported that the project had made a difference in their schools, including 
supporting the development of a more common vision and a more college-going 
culture. 


Methodology 


The evaluation design of the impact of SECEP in Connecticut experienced several significant 
challenges resulting in an inability to make any clear causal claims about impact. First, SECEP 
was implemented in three of the most disadvantaged high schools in the state, which made it 
impossible to identify comparison schools that were well matched on both outcome and 
demographic characteristics. Second, these schools were all located in a single district, which 
means that any program impact estimates would be confounded with effects from the district. 
Third, despite extensive efforts, we were unable to obtain access to student-level data or 
school-level data that were customized to meet the design requirements of our evaluation; as a 
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result, we used publicly available school-level data, which significantly limited the opportunities 
for statistical modeling and the types of outcomes we could examine. Finally, because the 
treatment groups were so homogeneous, we did not conduct any sub-group analyses. The 
results that we present below should be considered in the light of these substantial limitations. 


Sample 


The study sample in Connecticut included three treatment schools, all from Bridgeport. These 
schools were among the most disadvantaged in the state, with an average 99.7% poverty rate; 
only one other school in the state (which had broken into four small academies) had a similar 
poverty rate while the next closest was 14 percentage points lower. The treatment schools had 
an average of 89% minority enrollment rate. Nine schools (including the four small schools) had 
comparable minority rates but most of those schools (outside of the four small schools) were 
not similar on poverty. In terms of exploring baseline measures for outcomes, we were able to 
find schools that met WWC expectations for baseline equivalence for two of the outcomes 
measured—college credit coursetaking and dropout rates—but those schools were not 
comparable on demographic characteristics. We were unable to identify a set of schools fully 
comparable on baseline college preparatory coursetaking that were also somewhat similar in 
demographics. As a result, we decided to prioritize equivalence on the outcomes and then find 
as close a match as we could with demographic characteristics, recognizing that differences 
between the two groups of schools would not meet the WWC baseline equivalence standards. 
Despite the lack of equivalence on some measures, we believed, however, that the comparison 
schools would provide suggestive evidence regarding the impact of SECEP. Table VI-1 presents 
the baseline equivalence statistics for the three different samples used for the three different 
outcomes. 


Table VI-1. Baseline Equivalence, Treatment and Comparison Groups—Connecticut 


Treatment Control Standard Effect 
Ctrl Mean Deviation Size 
Outcome Measure N N (SD) (Pooled) (Hedge’s g) 
College Credit % College Credit Course Taking 3 9 16.73 6.75 07 
Course Taking (6.36) 
Sample “%Povery | | 9973 | | ¢ 63.39 | 866 | 3.87 
(9.68) 
“%Minority | | e932 | [4 72.50 | | 1534 | 101 
(17.00) 
Dropout Cohort Dropout rate 3 10 20.66 6.90 .02 


% Minority 83.25 9.52 59 
(10.30) 
Enrollment in % Enrollments in college 3 4 53.37 6.66 35 
college-ready | preparatory math and English Ine) ee 
math and English | % Poverty 60.68 11.49 2.86 
Bll  eeclitiN IL (1483) nf 
% Minority 65.31 11.40 1.77 
(14.23) 
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Outcomes 


All outcomes used publicly available school-level data, and, in cases of missing data, no 
outcomes or covariates were imputed. The following outcomes were examined: 


e Percentage of students enrolling in college credit-bearing or CTE courses. This 
measure, which was calculated by the Connecticut Department of Education and was at 
the school level, examined the percentage of enrolled 11°*- and 12""-grade students who 
took at least two AP, IB, dual enrollment, or CTE courses in the given school year. 
Outcome data are provided for the 2017-18 school year. It is important to note that CTE 
courses were not included in this measure in the baseline year, but they were included 
in 2017-18. 

e Cohort dropout rate. This outcome was the four-year cohort dropout rate, calculated by 
the Connecticut Department of Education. These data were examined for the 2017-18 
graduating cohort of students; most of these students would have been in 9" grade in 
2014-15, the year in which SECEP started. 

e Percentage of course enrollments in college preparatory courses. This outcome looked 
at the percentage of course enrollments taken by 9" graders that could be deemed as 
college preparatory. These courses include a college preparatory English course (English 
| or higher) and a college preparatory mathematics course (Algebra |, Algebra Il, 
Geometry, Precalculus or any AP mathematics course). This measure was calculated by 
the evaluation team using publicly available data that included the number of 
enrollments by grade level and by specific course. Outcome data were for the 2017-18 
school year. 


Analysis 


As above, the analyses were conducted using SPSS (version 25). Impact analyses on the three 
outcomes were conducted at the school level using a series of Analysis of Covariance (ANCOVA) 
models with the outcome as the dependent variable while controlling for baseline measures of 
the relevant outcome, poverty percentage, and racial minority percentage. 


Results 


The first outcome was the percentage of students taking college credit courses. As shown in 
Table VI-2, the SECEP schools and comparison schools provided similar levels of access to 
college credit/CTE courses (with estimated means of 67% for both). It is important to note that 
this school-level measure included CTE courses, which were not an emphasis of the 
intervention. Data collected by NCREST showed that dual enrollment/AP participation rates 
were approximately 15-17% in 2016-17, which suggests that the bulk of student participation 
was driven by CTE courses and not dual enrollment/AP courses. 
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As Table VI-2 also shows, SECEP expanded their enrollment in college preparatory courses at a 
higher rate than comparison schools when adjusted for demographic characteristics, although 
the difference was not statistically significant. Finally, dropout rates were higher in SECEP 
schools than in comparison schools by approximately five percentage points, although the 
difference was not statistically significant. 


Because of the small sample sizes, none of the differences between treatment and comparison 
schools were statistically significant. 


Table VI-2. Estimated Student Impacts, Connecticut 


Treatment 
Model adjusted Comparison Adjusted impact 

Outcome (Year measured) mean Unadjusted mean estimate p-value 
% 11" and 12" graders taking at least two college 66.7% 66.8% -.1% 99 
credit or CTE courses (17-18) 

Cohort dropout rate (16-17) 21.6% 17.0% 4.6% .25 
% 9'-grade course enrollments in college 87.9% 81.58% 6.4% 71 
preparatory courses (17-18) 


Note. The treatment mean was calculated by adding the impact estimate from the model to the unadjusted comparison mean; essentially, the 
model estimates what the mean for the treatment schools would be if they had the same levels of poverty and race/ethnicity as the 
comparison schools. No differences were statistically significant. 


In considering these results, it is important to remember that the treatment and comparison 
schools were not fully comparable. Even when the schools started with similar levels of 
outcomes, the treatment schools had higher poverty rates and higher percentages of minority 
students. These findings suggest that schools in challenging environments can still make 
changes. There was an increase in dropout rates in treatment schools over the life of the 
project, which is concerning and should be an area of attention for the district moving forward. 


Perceptions of Impacts of the Program on Staff and Students 


In addition to looking at outcomes using school-level administrative data, we also explored 
stakeholder perceptions of the impact of the program using multiple sources of data, including 
the staff survey and site visits. We begin by presenting the findings from the impact survey and 
then share how staff and students discussed the impact of the program. 


Survey Results 


On the staff survey, we asked participants to indicate the extent to which they agreed with a 
set of statements around the potential impacts of the program on school culture and students. 
Response options ranged from “strongly disagree” (1) to “strongly agree” (4). Table VI-3 
presents the percentage of respondents who “agreed” or “strongly agreed” with six 
statements. The results are broken out by role type (i.e., all program teachers, STEM program 
teachers, and administrators). 
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Table VI-3. Perceptions of Impact on School Culture and Students, Connecticut 


Fourth Survey (2018) 
Agree + 
Strongly Agree 
Item Respondent n (%) Mean 
Because of SECEP, the staff in our school 
have higher expectations for all students. 


Because of SECEP, we see positive changes 
in student engagement, academics, or 
interest in STEM areas. 


Program Administrators 
Al 


Because of SECEP, there is an increased 
collaboration among teachers. 


Program Administrators 
Al 


Because of SECEP, trust and collaboration 
among all staff has improved. 
Program Administrators 1S 80.0 3.07 
Al 


Because of SECEP, our school is more 

focused on a common vision and mission. 
Program Administrators 16 81.3 3.06 
Because of SECEP, relationships between All Program Teachers 82 70.7 2.71 
teachers and students have improved. 


Program Administrators 


A majority of respondents “agreed” or “strongly agreed” that SECEP had resulted in impacts on 
school culture and on students. The highest perceived impact was in the area of developing a 
common vision and mission for the school, followed by increased expectations for students and 
higher student engagement. Over 70% of all teachers and 80% of administrators “agreed” or 
“strongly agreed” that SECEP had resulted in positive changes in student engagement, 
academics, or interest in STEM areas. The area with the lowest agreement in terms of perceived 
impact for administrators (80.0%) and teachers (68.8%) was improvement of trust and 
collaboration among all staff. 


Site Visit Results 


According to interviews, the SECEP project has led to impacts and changes at both the staff and 
student levels in Bridgeport. 


Impact on School Culture 


Ten staff from all four of the schools we visited noted that the grant has given a common focus 
to the schools that has resulted in increased communication and collaboration among teachers 
and, for three of the schools, between teachers and administrators. As one teacher noted, 


With our i3 meetings, | see a lot more collaboration between grade levels. And then also 
between content areas. So, | would not have been exposed to talking to elective class 
teachers about instructional strategies and observing their classes, or | would not have 
so much conduit to 10°"-grade teachers who are teaching the students we had last 
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year.... Just having that common planning time and having those meetings helps us to 
talk about [what | did] last year, so they know where they're coming from. 


One district staff member, however, did express concerns that, because the collaboration was 
driven primarily by participation in project activities, these changes might not last beyond the 
grant. 


Representatives from three of the four schools believed that there was an increase in teacher 
leadership, primarily centered around sharing results from the project professional 
development and in taking more ownership around selecting professional development and, in 
one school, supporting instructional rounds. As a school leader described, 


What we've seen is that some teachers have really stepped up into making sure that 
they had their say as to how this is implemented; sharing ideas, planning, as well as 
debriefing, and one of the large areas that we've done and expanded on...is to go to the 
classroom rounds where teachers are going around and observing each other's lessons 
and then giving feedback to each other and, as administrators, we've taken a step 
back.... It's a different experience when the administrators aren't involved directly in this 
and we've allowed them to do this. 


In one school, the principal noted that the grant has increased expectations of the leadership 
team to become more instructional leaders. 


Impact on Students 


Relative to student experiences, staff in both high schools commented that students were 
paying increased attention to what happens after high school. In one school, the staff described 
their school as having more of a college-going culture. In the other high school, two staff and 
two students noted that there was more of an orientation around one’s future among the 
students. As a faculty member said, 


One huge change from my first year...is that, not only are the classrooms full, but 
students are interested in learning and they want to have more knowledge, and even 
though | have sophomores, they are already focused on, “When | leave high school, | 
want to study this...” or they have an idea of having a future, where some of the 
students | had two years ago couldn't think that they would even graduate let alone to 
think about the next step past graduation. 


Overall, the grant was perceived to afford more opportunities for students to take college 
courses. 
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Staff from all four schools also believed that teachers were implementing the targeted 
instructional practices and that those practices were leading to increased student engagement. 
One teacher commented, 


I've seen students become more involved and I've been able to design activities where 
they can show me what they know, where they're able to do the work independently, 
and so they are drawing their own conclusions, making their own connections, and I've 
seen much more meaningful learning that way. 


Staff from two schools noted that the changing instructional practices have also led to 
improved teacher-student relationships and improved student attendance. One teacher said, 


| just feel like the relationships within the building have changed so much in the past 
two years. | remember times when my first period class would have three people in it, 
now, by the first bell, rarely are there less than 20. We had 400 students, | think it was, 
in the first marking period with perfect attendance, which is almost 40% of our 
population and, to me, a great number of students that [show] up every day and really 
care about their education. 


Faculty from three of the schools reported that student achievement was rising because of 
increased student engagement with the content. One teacher noted, “With the strategies and 
projects and making it more student-centered...it's great to see those light bulbs go off because 
they discovered it. Then I've found that because they discovered it, they remember it.” 


As previously mentioned, the Connecticut study design had substantial challenges, thus, we 
cannot make any causal claims about the impact of the intervention. Overall, however, results 
suggest that the Bridgeport schools made some changes because of SECEP but that those 
changes were not sufficient to increase outcomes above and beyond what other schools in the 
state were already doing. It is important to remember that the SECEP schools faced 
considerable challenges in terms of poverty and historically low performance. Additionally, the 
descriptive increase in cohort dropout rates for treatment schools remains an area of concern. 
Although we are not convinced that the intervention caused an increase in dropout rates, the 
findings suggest that dropout prevention should be a key focus area for the district moving 
forward. 
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SECTION VIII: DISCUSSION AND CONCLUSION 


SECEP was the first large scale effort to take lessons learned from the successful small Early 
College Model, merge them with a STEM focus, and apply them in comprehensive high schools. 
Project staff executed a key set of implementation support strategies—including technical 
assistance, coaching, professional development, management teams, an online COP, and 
support to high school-college partnerships—that were intended to assist schools in 
implementing the four STEM Early College Design Principles. 


The effort was designed to increase the number of students completing high school with at 
least one college credit and to better prepare students in STEM. Past studies have suggested 
that implementing change in comprehensive high schools can be a challenging task (American 
Institutes of Research & SRI International, 2008; Edmunds et al., 2018; Mazzeo, Fleischman, 
Heppen, & Jahangir, 2016). Results from this evaluation suggest that schools can change their 
practices in ways that improve student outcomes but that there are also issues that need to be 
considered in moving this work forward. 


Changes in Schools 


Survey and site visit data showed that participating schools did make changes in 
implementation of the Design Principles, particularly related to a STEM College-Focused 
Academic Program, High School-College Collaboration, and a Culture of Continuous 
Improvement. 


STEM College-Focused Academic Program 


The schools reported implementing more college courses, developing STEM pathways, 
increasing college readiness activities and changing instructional practices. In Michigan, each 
district had at least one Early/Middle College, which provided extensive access for the students 
enrolled in that school. Across both states, schools had students enrolled in STEM-focused 
pathways that spanned high school and college often in engineering, information technology, 
medical fields, or advanced manufacturing areas. On the surveys, administrators reported 
significant increases in the percentage of students on track to earn college credits while in high 
school. 


In addition to expanding access to college courses, all middle and high schools in both states 
focused on academic preparedness and college and career awareness and aspirations. High 
schools in both states offered a variation of a college readiness course, which focused on 
academic college skills (soft skills) and college knowledge. Some schools taught these skills in a 
separate class whereas others embedded them in regular subject classes. Students in both 
states found these classes helpful. 
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Across both states, there have been reported changes in instruction in all schools. In 
Bridgeport, schools implemented the CIF strategies, and in Michigan, schools focused on 
implementing project-based and inquiry learning. While all teachers we interviewed 
implemented new instructional strategies, the survey results suggest that not all teachers in the 
schools had embraced the changes. 


Student Support 


Across both states, schools increased the supports provided to students taking college courses, 
providing additional tutoring or monitoring students’ performance in college classes. There was 
also an increase in collaboration with the colleges around supports. There was no indication, 
however, that schools made substantial changes in the types of supports they provided relative 
to high school courses. While academic and postsecondary support appeared to be more 
formal, there seemed to be a less systematic approach for affective support (except for one 
emotional support intervention in Bridgeport that was not part of SECEP although it was seen 
as complementary). 


High School-College Collaboration 


In both Bridgeport and Michigan, it was evident that the grant had increased the number of 
postsecondary partners and strengthened ongoing partnerships. Across both states, there was 
a clear expectation that this would be one of the most sustainable parts of the grant. A staff 
member in Michigan said, “It is going to be a great partnership because we're so intertwined.” 


Culture of Continuous Improvement 


Schools increased their use of data as part of the SECEP project, primarily focused on college 
courses. Schools also reported using and appreciating the range of data provided by NCREST 
through its data analysis and training. Teacher collaboration was also a component of 
continuous improvement; there was disagreement here, however, between the interview and 
the survey data. The survey showed no changes in collaboration over time, while the interviews 
indicated that there was much more collaboration, at least, in part, because teachers were 
trying to implement more interdisciplinary projects with their colleagues. 


Impacts on Students 


As described above, the comprehensive schools in this project made changes to their schooling 
environment, increasing the emphasis on college-going and working to implement changes in 
instructional practices. The changes made by the schools were expected to lead to improved 
student outcomes. To test these hypotheses, the evaluation study had to conduct two separate 
impact studies, broken out by state. This turned out to be appropriate and necessary given that 
the context and implementation varied substantially by state. Due to the challenges with the 
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Connecticut impact analyses, however, we focus this discussion primarily on the results from 
Michigan. 


In the Michigan schools, we saw an overall positive impact on students’ earning college credits. 
The percentage of students who earned at least one college credit was 12 percentage points 
higher in the SECEP schools than in the comparison schools (29.8% vs. 17.5%). Students in 
SECEP schools earned approximately double the number of college credits as students in 
comparison schools (3.5 vs. 1.8). 


Evidence from the Michigan impact study suggests that enrollment in dual credit courses may 
have come somewhat at the expense of enrollment in AP courses. For example, the percentage 
of students enrolled only in dual credit courses was 13 percentage points higher than in the 
comparison schools, while the percentage enrolling only in AP courses was nine percentage 
points lower. However, study results also suggest that enrollment in dual credit courses made it 
more likely that a student would earn potential college credit. We found a very small 
proportion of our sample scoring sufficiently high on AP exams to earn college credit while over 
90% of the students taking dual credit courses received a passing grade. Of course, it is 
important to acknowledge that earning actual credit, whether through dual enrollment courses 
or AP exams, always depends on the college or university granting that credit. 


There is also evidence from Michigan that dual enrollment was not reaching all students 
equally, particularly with regard to minority students. The project impacts were smaller for 
minority students and minority students were less likely to enroll in dual enrollment courses 
than non-minority students. These findings are consistent with other literature that has shown 
that expansion of dual enrollment opportunities often comes first to white or more advantaged 
students, resulting in concerns about equity (Miller et al., 2018; Pierson, Hodara, & Luke, 2017). 
These data suggest that schools will want to regularly look at the characteristics of their dual 
enrollment students to ensure that there is equity in their coursetaking opportunities. 


The heightened college-going culture was intended to give students a better sense of their 
future and encourage them to remain in school. The dropout data showed descriptively lower 
dropout rates in SECEP schools, although the difference was not statistically significant. 


Overall, results from the SECEP study show that comprehensive high schools can make 
significant changes to their culture and their instruction. The hope is that these changes will 
lead to more students graduating from high schools prepared to enroll in, and be successful in, 
careers or further postsecondary education. 
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APPENDIX A: SURVEY 


Summary of Survey Scales’ Properties and Questions 


Staff Scale a 


Questions on the Scale (T — teachers only, A— administrators only, T, A —all 
staff) 


Response Options 


SECEP Program .88 
Understanding and 
Buy-in (T, A°) 


- | understand the goals and objectives of the SECEP program that is being 
implemented in our school. (T, A) 

- Most staff in our school understand the goals and objectives of SECEP 
program. (T, A) 

- Ateam of staff and administrators participated in developing goals for 
effective STEM Early College education in our school. (T, A) 

- | believe that our school should prepare students for postsecondary education 
and careers, with an emphasis on STEM fields. (T) 

- | have made focused efforts to align my lessons with college expectations. (T) 

- All staff in this school share a goal of preparing every student for 
postsecondary education without remediation. (T, A) 

- Teachers make focused efforts to align their lessons with college 
expectations. (A) 

- lam aware of our school’s academic plan or scope & sequence that shows 
students how they can earn college credits by high school graduation. (T, A) 

- All staff in this school share a goal of preparing students for STEM-focused 
postsecondary education or careers. (T, A) 

- lam aware of our school’s pathways or sequences of high school and college 
courses leading to STEM college majors. (T, A) 

- Our school has a school-wide initiative to improve STEM teaching strategies. 
(T, A) 

- All staff are committed to providing necessary supports to ensure that 
students are prepared for postsecondary education and careers with an 
emphasis on STEM. (T, A) 


College-going 81 
Expectations (T, A) 


College Readiness 84 
(T, A) 


- Most faculty and staff in this school expect every student to receive at least 
some postsecondary education or training. 

- Most faculty and staff in this school believe that, if given enough support, all 
students can successfully complete college preparatory courses. 

- Our school explicitly and purposefully focuses on building students’ 
postsecondary aspirations. 

- Our school explicitly and purposefully focuses on helping students to identify 
their future career directions. 


as content knowledge, writing, reading, and critical thinking skills, etc.). 

- The school shared these goals for academic college readiness with all staff. 

- There are school-wide rubrics (or other guidelines) staff can use to measure 
academic college readiness of their students. 

- Staff incorporates measures of academic college readiness into their 
classroom. 


School Academic 85 
and Social- 
Emotional Support 
(T, A) 


Our school has a system to identify students in need of academic support. 
Our school has a system to identify students in need of social/emotional 
support. 

| am aware of supports that the college provides to students taking college 
courses. 

Our school has a system to provide support to students taking college 
courses. 

The high school and college collaborate on providing support to students in 
college courses. 

- The school makes efforts to increase family engagement in students’ 
academic life. 

Faculty or staff members follow up when students miss their classes. 


1 = Strongly Disagree 
2 = Disagree 

3 = Agree 

4 = Strongly Agree 


1 = Not True at All 
2 = Somewhat True 
3 = Mostly True 

4 = Entirely True 


- The school has clearly articulated goals for academic college readiness (such 


5 “T” indicates teacher items and “A” indicates administrator items. 
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Staff Scale 


Questions on the Scale (T — teachers only, A— administrators only, T, A —all 
staff) 


Response Options 


Staff Academic and 
Social-Emotional 
Support (T, A) 


78 


- | regularly provide academic supports to students who need it. 

- | regularly provide social and/or emotional supports to students who need it. 

- Most staff regularly provide academic supports to students who need it. 

- Most staff regularly provide social and/or emotional supports to students 
who need it. 


College Readiness 
Skills 


76 (T) 
92 (A) 


(A) How frequently have most of your students received explicit training in the 

following areas: 

- Time management 

- Note taking 

- Organizational skills 

- Advocating for themselves with high school and college faculty. 

(T) In the current school year, how often have you: 

- Worked with students on time management and study skills? 

- Worked with students on organizational skills? 

- Expected students to take detailed notes on a lecture or presentation? 

- Encouraged students to advocate for themselves with high school and 
college faculty? 


Rigorous 
Instruction 


84 


(T) In the current school year, how often have you: 
(A) How frequently have most of your teachers implemented the following 
instructional practices? 
- Encouraged students to find more than one way to answer a question? 
- Had students develop and analyze multiple solutions to a problem? 
- Asked students to defend their own ideas or point of view in writing or ina 
discussion? 
- Asked students to explain their thinking? 
- Asked students to engage in in-depth discussions about what they have read 
or learned? 
- Asked students to analyze or interpret documents or data? 


Inquiry, Project- 
Based Learning 
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(T) In the current school year, how often have you: 
(A) How frequently have most of your teachers implemented the following 
instructional practices? 
- Asked students to solve problems based on life outside of school? 
- Asked students to develop and/or test a theory or hypothesis? 
- Had students develop their own questions and then answer them? 
- Implemented projects in their classrooms? 
- Asked students to do a formal oral presentation? 
- Had students collaborate on projects or assignments? 
- Asked students to read subject-related texts that are difficult or complex? 
- Asked students to write subject-related essays longer than 5 pages? 


1=Never 

2=A Few Times This 
Year 

3=Once or Twice a 
Month 

4=Once or Twice a 
Week 

5=Almost Every Day 


High School and 
College 
Collaborations (T, 
A) 
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This school year, how often did you meet with college faculty: 
- to discuss student support. 

- to understand college expectations in my subject area. 

- to improve student opportunities to study in STEM areas. 
- to establish curriculum alignment. 

- For joint professional development with college faculty. 


1 = Never 
2=1-2Times 
3=3-6 Times 
4 = More Than 6 
Times 


Local College 
Support (T, A) 


.89 


To what extent have you or your school received the following 
resources/supports from your local college over the past year? 
- Financial resources 

- Access to college facilities 

- Guest speakers 

- Curriculum materials 

- Research or internship opportunities 

- College awareness materials 

- Professional development 

- College courses 


1=None 

2 =Alittle 

3 =A fair amount 
4=Alot 
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APPENDIX B: INTERVIEW PROTOCOLS 


Table B-1. Interview Questions Crosswalk Across Respondents 


Construct 


HS 
Princip 


HS 
Teach 


MS Ms 
Princip | Teach 


College 
Staff 


Coach 


Total number of questions (not including surveys) 


20 


16 


17 16 


15 


Focus of 
work 


What was the focus of your work related to the 
SECEP project in the current school year? 


STEM college-focused academic program 


What instructional coaching have you 
received/provided during this school year? 
Please describe your school’s efforts over the past 
two years to expand access to college courses for 
more students and proportion of students 
Currently impacted by these efforts. | 
Please describe your school’s efforts over the past 
two years to improve students’ college readiness 
skills and proportion of students currently 
impacted by these efforts. 
Please describe your school’s efforts over the past 
two years to create STEM and other pathways that 
span high school and college courses and 

proportion of students currently impacted by these 
OFFS. 
Please describe your school’s efforts over the past 
two years to change STEM instruction, and 
instruction in other subjects and proportion of 


students currently impacted by these efforts. 


Academic and social 


supports to students 


Please describe your school’s efforts to expand 
academic and social supports to your students as a 
result of this project and proportion of students 
currently impacted by these efforts. 


Please describe how the partnership with your 
postsecondary partner(s) has evolved during the 
past year. 


Use of data to 


improve student | Partnerships 


outcomes 


To what extent are teachers provided time for 
common planning and collaboration? | 
How are you using data differently as a result of 
this project? As a probe: do you share data 


between high school and college? 


Impacts 


Other than what you have described above, how 
has this project impacted any aspects of your 
school, teachers’, or your own work over the past 
four years? 


What are the impacts of the SECEP program on 


p p y p 
project on organizational culture shift. The next 
question refers to this survey.) 
Would you say that the project resulted in some 

shifts in organizational culture, and if so, in which 


What do you see as impacts of this project on 


your community college? 
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HS HS MS MS Distr. Coun- | College 


Construct Princip | Teach | Princip | Teach Staff selor Staff Coach 
Total number of questions (not including surveys) 20 16 ail7/ 16 16 16 15 17 
What kinds of support did you receive from 
MCNC/JFF, district or school staff in implementing X xX X X X 


activities related to SECEP? 


How useful was the support? What other kinds of 
support would you have liked to have? 


How does the school monitor implementation of 
the SECEP Design Principles? Did you use the X Xx 
_SECEP Design Principles Self-Assessment ea enemies 
To what extent are teachers expected to 
implement the new strategies? To what extent is xX X xX 
_Whieir Instruction monitored far TESe Pract Ce tal tenant kneel eat tial acsleltie Mieatieindl 
What role did monitoring and accountability play 
in the implementation of SECEP? (Probes: 
monitoring by the project, by the district, by the 
principal). 
[Participants are asked to respond to a brief 
survey evaluating the capacity for sustainability of 
Design Principles. The next questions refer to this 
A 2 DO 
Please comment on your responses for how likely 
are various components of SECEP to be sustained X xX X X X X X X 
SESE Shs a me ee | See: er (| REED | cen EE me | eee 


What features of SECEP were most likely to be 
supported by school leaders and teachers and X X 


Support and Monitoring 


Are you convinced that the STEM Early College 
approach is worth implementing? If so, what X xX xX X xX X 
convinced you? 

Looking back at the SECEP implementation, which 
were the most effective strategies used to support x X X x x x xX X 
implementation of the program? 


Buy-in and Sustainability 


i= 

| Cn enn a ne er eETE EN E| RPDS) ESERIES [SCRE (EOE EERE CSCO (OREN 

S Were there specific implementation strategies that 

5 you would not recommend using if you were going X X xX X xX X xX X 

“| to implement this program in the future? 
Is there anything else you’d like to tell us? X xX xX X X X X xX 
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APPENDIX C: COMPLETED PATHWAYS 


Pathways Created or Revised Over the Life of SECEP Project 


District 
Bridgeport 


Pathways 
Bassick 
- Advanced Manufacturer 
- Revised Business Academy 
- Revised FAME Academy 
- Revised Sci Tech Academy 
- Healthcare Career and College Readiness Program 


Central 

- Business Management and Hospitality 

- Arts and Media 

- Revised Government, Public Administration, & Safety 


Harding 

- Revised IB Law Careers Academy 

- Revised Culinary Arts 

- Revised Health Careers Magnet Academy 


Delta Schoolcraft ISD 
- Escanaba 
- Bark River 


Genesee 
- Carman Ainsworth 
- Clio 


AA, AAS and Certificate pathways: 

- Arts and Design 

- Business Administration 

- Automotive Technology 

- Computer Network Systems and Security 

- Environmental Management 

- Water Resource Management 

- Mechanical Engineering 

- Pre-Professional Health 

- Welding Certificate 

- Transfer Student Generic 

- Engineering 

- Elementary Education — Mathematics/Integrated Sciences Minor 
- Elementary Education — Language Arts 

- (Bark River & Escanaba) Early Childhood Education 
- Automotive Pathway 

- Welding Pathway 

Carman Ainsworth Early Middle Program @ Baker College 
- Automotive Services 

- Criminal Justice 

- Advanced Manufacturing 

- Photonics and Laser Technology 

- Computer Programming 


Carmen Ainsworth STEM Early College @ University of Michigan-Flint 
- Medical Sciences Pathway 

- Pre-engineering Pathway 

- Health Sciences 


Clio (DEEP enrollees) @ University of Michigan-Flint 
- Medical Science Pathway 

- Health Sciences 

- Engineering 

- Law 


Clio & Carmen-Ainsworth/Genesee County Career and Technical Education Early Middle College 


- Computer Occupations Technology — Applications Developer 
- Computer Occupations Technology — Computer Security 

- Nursing, AAS Prerequisites 

- Automotive Technology 

- Industrial & Manufacturing Technology AAS 


Lapeer 
- Almont 
- Dryden 


Almont, Imlay and North Branch 
- 5-year STEMM Academy Plan @ Baker College 
- 5-year CTE Blend @ Baker College 
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District 


Pathways 


- Imlay City 
- North Branch Area 


Washtenaw 
- Ypsilanti 


Dryden (same pathways as the other 3 districts, submitted form with a different title) 
- Early College 

- STEMM Early College Technical Track 

- STEMM Early College Medical Track 


Almont, Dryden, Imlay and North Branch/STEMM Academy (Early/Middle College—Science, Technology, 
Engineering, Mathematics & Medical) & Baker College of Flint 

- Engineering/Technology Programs 

- Health Science Programs 

- Nursing Programs 


- Automotive Services Technician Certificate 

- Collison Repair Certificate 

- Baking & Pastry Certificate 

- Culinary Arts Certificate 

- Child Development Associate Certificate 

- Health Care Foundations Certificate (2 excel sheets) 
- Introduction to Manufacturing Processes Certificate 
- Machine Tool Set-Up and Operations 

- Fluid Power (Pre-Mechatronics) 

- Machine Tool Programming (Pre-Mechatronics) 

- Industrial Electronics Technology 

- Welding & Fabrication Principles Certificate 

- Computer Systems Technology Certificate 

- MEMCA Certificate 

- Michigan Transfer Agreement Certificate 

- Aviation Airframe & Powerplant Technician Certificate 
- Energy Technician Certificate 
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APPENDIX D: BASELINE EQUIVALENCE BY SUB-GROUP, MICHIGAN 


Table D-1. Baseline Equivalence, Treatment and Comparison Groups, Michigan--Minority 


Treatment Comparison 
Mean Mean Standard 
(SD) (SD) Deviation Effect 
Measure [IN] [IN] (Pooled) Size 
Cohort dropout rate .022 .013 
(.14) (.11) NA 33 
[2,033] [7,961] 
% of 11'- and 12**-grade students receiving at least one .015 .015 
college credit (.083) (.121) N/A .02 
[864] [3,471] 
Average college credits earned .169 .091 07 
(1.23) (1.06) 1.10 , 
[864] [3,471] 
% Economically Disadvantaged 717 .702 
(.458) (.439) N/A .05 
[2,033] [7,961] 


Table D-2. Baseline Equivalence, Treatment and Comparison Groups, Michigan—Non-Minority 


Treatment Comparison 
Mean Mean Standard 
(SD) (SD) Deviation Effect 
Measure [IN] [IN] (Pooled) Size 
Cohort dropout rate .008 .009 
(.095) (.101) N/A 013 
[4,534] [17,411] 
% of 11'- and 12*-grade students receiving at least one .086 .087 
college credit (.274) (.282) N/A 01 
[2,157] [8,398] 
Average college credits earned 565 .605 
(2.16) (2.56) 2.48 02 
[2,157] [8,398] 
% Economically Disadvantaged 372 .403 
(.491) (.487) N/A .08 
[4,534] [17,411] 


Table D-3. Baseline Equivalence, Treatment and Comparison Groups, Michigan—Economically 


Disadvantaged 
Treatment Comparison 
Mean Mean Standard 
(SD) (SD) Deviation Effect 
Measure [IN] [IN] (Pooled) Size 
Cohort dropout rate .018 .017 
(.14) (.13) 03 
[3,254] [12,644] 
% of 11'- and 12**-grade students receiving at least one .031 .028 
college credit (.160) (.165) N/A 05 
[1,374] [5,396] 
Average college credits earned .238 .206 
(1.30) (1.55) 1.50 .02 
[1,374] [5,396] 
% Minority .506 438 
(.496) (.499) N/A 17 
[3,254] [12,644] 
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Table D-4. Baseline Equivalence, Treatment and Comparison Groups, Michigan—Not 
Economically Disadvantaged 


Treatment Comparison 
Mean Mean Standard 
(SD) (SD) Deviation Effect 
Measure [N] [N] (Pooled) Size 
Cohort dropout rate .008 .005 
(.074) (.070) N/A .26 
[3,313] [12,728] 
% of 11'*- and 12**-grade students receiving at least one .100 .097 
college credit (.284) (.296) N/A .02 
[1,647] [6.473] 
Average college credits earned .686 .661 
(2.34) (2.66) 2.60 01 
[1,647] [6.473] 
% Minority .220 .184 
(.387) (.369) N/A 14 
[3,313] [12,728] 
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APPENDIX E. SUB-GROUP IMPACTS, MICHIGAN 


Table E-1. Impacts for 11" Graders, Michigan 


Treatment | Comparison 
(N=3,866) (N=15,272) 
Adjusted Impact Impact 
Adjusted Unadjusted Estimate at Estimate 
Outcome Mean (SD) Mean (SD) Year 4 Standard Error _ p-value 
% enrolled in any college 35.45 28.70 
course (45.27) (45.24) ide ad aes 
% 11" enrolled in dual 15.58 4.54 
enrollment courses only (33.53) (20.83) anes a7 GUS 
% enrolled in AP courses 15.73 22.37 
Gtily (31.95) (41.68) -6.64 4.40 0.138 
% enrolled in both AP and 4.25 1.78 
dual enrollment courses (20.33) (13.23) iad ae ou 
% receiving at least one 21.04 11.42 
college credit (39.51) (31.81) ae eee said 
% receiving at least one 
a 16.25 5.71 
college credit—dual (36.68) (23.20) 10.54 3.36 0.003 
enrollment only 
% receiving at least one 4.03 5,32 
college credit—AP only (15.88) (22.45) “Aa £98 oals 
% receiving at least one 
college credit—both AP and nae ies 0.33 0.34 0.336 
(8.63) (6.20) 
dual enrollment 
# college credits earned 1.77 0.89 
(4.12) (3.26) 0.88 0.29 0.003 
# college credits earned— 1.37 0.46 
dual enrollment only (3.91) (2.26) gt OS 08 
# college credits earned — 0.39 0.42 
AP only (1.28) (2.39) ae ot nen 
Table E-2. Impacts for 12 Graders, Michigan 
Treatment | Comparison 
(N=3, 706) (N=14,815) 
Adjusted Impact Impact 
Adjusted Unadjusted Estimate at Estimate 
Outcome Mean (SD) Mean (SD) Year 4 Standard Error | p-value 
% enrolled in any college 48.98 41.88 
course (49.90) (49.34) 0 a Oa 
% 11* enrolled in dual 21.60 8.73 
enrollment courses only (36.11) (28.23) i287 om slau 
% enrolled in AP courses 18.85 28.15 
only (40.24) (44.97) sie ae ve 
% enrolled in both AP and 8.69 5.00 
dual enrollment courses (31.27) (21.80) i ane oes 
% receiving at least one 31.57 20.26 
college credit (44.76) (40.19) ahee aot aye 
% receiving at least one 
: 23.91 11.91 
college credit—dual (41.31) (32.39) 12.01 4.39 0.009 
enrollment only 
% receiving at least one 4.84 7.24 
college credit—AP only (16.67) (25.92) eld il claw 
5 oe 
ee eee mle 
8 (17.12) (10.46) : : : 
dual enrollment 
# college credits earned 4.17 2.13 
(9.21) (6.16) 2.04 0.51 0.000 
# college credits earned— 3.50 1.38 
dual enrollment only (8.96) (5.04) 2 ne? neue 
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Treatment | Comparison 
(N=3, 706) (N=14,815) 
Adjusted Impact Impact 
Adjusted Unadjusted Estimate at Estimate 
Outcome Mean (SD) Mean (SD) Year 4 Standard Error | p-value 
# college credits earned — 0.75 
AP only (1.71) (3.65) -0.09 0.22 0.687 


Table E-3. Impacts for Minority Students, Michigan 


Treatment Comparison 
(N=1,527) (N=6,694) Adjusted Impact 
Adjusted Mean Unadjusted Estimate at Impact Estimate 

Outcome (SD) Mean (SD) Year 4 Standard Error p-value 
% enrolled in any college course 34.20 33.58 

(43.40) (47.23) 0.62 5.44 0.910 
% 11* enrolled in dual enrollment 12.08 3.84 

8.24 3.47 0.023 

courses only (30.81) (19.22) 
% enrolled in AP courses only 19.61 26.73 

(31.71) (44.26) -7.12 5.39 0.193 
% enrolled in both AP and dual 2.95 3.02 
enrollment courses (17.63) (17.11) ian aa hia 
% receiving at least one college 14.68 8.90 
credit (33.96) (28.48) ae a0 tee 
% receiving at least one college 12.95 5.84 
credit—dual enrollment only (32.79) (23.45) yet ae one 
% receiving at least one college 0.94 2.49 
credit—AP only (8.83) (15.60) “tee 108 oie 
a oa 
% receiving at least one college 051 0.57 
credit—both AP and dual -0.06 0.54 0.913 

(5.11) (7.51) 
enrollment 
# college credits earned 1.31 0.78 

(4.41) (3.75) 0.53 0.47 0.263 
# college credits earned—dual 1.16 0.59 
enrollment only (3.42) (1.32) sae a ain 
# college credits earned —AP only 0.10 0.19 

(0.57) (1:52) -0.09 0.09 0.362 


Note. A careful reader may wonder why the % enrolled in dual enrollment courses only is lower than the % receiving at least one college credit 
only through dual enrollment. This is because individuals who are receiving credit only through dual enrollment may also have been 
simultaneously enrolled in AP courses (and thus show up in the percentage of students enrolled in both AP and dual enrollment courses) but 
did not receive credit from those AP courses; for that reason, they would then be included in the % receiving credit only from dual enrollment 
courses, even though they were enrolled in AP courses. 
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Table E-4. Impacts for Non-Minority Students, Michigan 


Treatment Comparison 
(N=3,703) (N=14,169) Adjusted Impact 
Adjusted Mean Unadjusted Estimate at Impact Estimate 
Outcome (SD) Mean (SD) Year 4 Standard Error p-value 
% enrolled in any college course 45.95 39.35 
(50.01) (48.86) 6.59 4.99 0.193 
% 11* enrolled in dual enrollment 20.89 9.15 
11.75 4.75 0.017 
courses only (37.75) (28.83) 
% enrolled in AP courses only 16.39 25.73 
(39.89) (43.71) -9.34 5.50 0.097 
% enrolled in both AP and dual 8.63 4.48 
4.15 2.25 0.072 
enrollment courses (33.20) (20.69) 
% receiving at least one college 32.29 21.53 
credit (46.65) (41.10) ceed i ene 
% receiving at least one college 23.65 11.91 41.74 425 0.008 
credit—dual enrollment only (43.37) (32.40) , , 
% receiving at least one college 5.93 8.53 
credit—AP only (18.41) (27.94) cha A Dass 
i =a 
rea are ria 2.92 1.08 1.84 on 0.013 
(18.06) (10.34) : , , 
enrollment 
# college credits earned 3.96 2.21 
1.75 0.45 0.000 
(9.09) (6.09) 
# college credits earned—dual 3.23 1.35 
enrollment only (8.83) (4.82) 188 oot eon 
# college credits earned —AP only 0.72 0.87 
(1.86) (3.85) -0.14 0.24 0.549 


Note. A careful reader may wonder why the % enrolled in dual enrollment courses only is lower than the % receiving at least one college credit 


only through dual enrollment. This is because individuals who are receiving credit only through dual enrollment may also have been 


simultaneously enrolled in AP courses (and thus show up in the percentage of students enrolled in both AP and dual enrollment courses) but 


did not receive credit from those AP courses; for that reason, they would then be included in the % receiving credit only from dual enrollment 


courses, even though they were enrolled in AP courses. 
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Table E-5. Impacts for Economically Disadvantaged Students, Michigan 


Treatment Comparison 
(N=2,462) (N=9,478) Adjusted Impact 
Adjusted Mean Unadjusted Estimate at Impact Estimate 
Outcome (SD) Mean (SD) Year 4 Standard Error p-value 
% enrolled in any college course 35.51 28.14 
7.37 5.23 0.165 
(46.17) (44.97) 
% 11" enrolled in dual enrollment 17:78 5.77 
courses only ; (23.32) 12.01 3.43 0.001 
(32.67) 
% enrolled in AP courses only 12.76 20.01 
-7.25 5.04 0.157 
(34.12) (40.01) 
% enrolled in both AP and dual 5.04 2.35 
2.69 1.29 0.044 
enrollment courses (22.21) (15.16) 
% receiving at least one college 20.69 10.17 
‘ 10.52 2.95 0.001 
credit (37.62) (30.23) 
% receiving at least one college 18.42 7.28 
7 11.14 2.97 0.000 
credit—dual enrollment only (35.66) (25.98) 
% receiving at least one college 1.14 2.54 
3 -1.40 1.03 0.180 
credit—AP only (10.97) (15.74) 
3 ar 
% receiving at least one college 1.09 0.35 
credit—both AP and dual 0.74 0.45 0.102 
(9.41) (5.89) 
enrollment 
# college credits earned 2.30 0.95 
-1.35 0.30 0.000 
(6.17) (3.99) 
# college credits earned—dual 2.16 0.73 
-1.43 0.32 0.000 
enrollment only (6.05) (3.52) 
# college credits earned —AP only 0.14 0.21 
0.07 0.10 0.488 
(0.82) (1.86) 


Note. A careful reader may wonder why the % enrolled in dual enrollment courses only is lower than the % receiving at least one college credit 


only through dual enrollment. This is because individuals who are receiving credit only through dual enrollment may also have been 


simultaneously enrolled in AP courses (and thus show up in the percentage of students enrolled in both AP and dual enrollment courses) but 
did not receive credit from those AP courses; for that reason, they would then be included in the % receiving credit only from dual enrollment 


courses, even though they were enrolled in AP courses. 
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Table E-6. Impacts for Non-Economically Disadvantaged Students, Michigan 


Treatment Comparison 
(N=2,768) (N=11,385) Adjusted Impact 
Adjusted Mean Unadjusted Estimate at Impact Estimate 
Outcome (SD) Mean (SD) Year 4 Standard Error p-value 
% enrolled in any college course 51.99 45.30 
6.69 5.15 0.200 
(49.92) (49.78) 
% 11" enrolled in dual enrollment 21.14 8.84 
° 12.30 5.30 0.025 
courses only (38.48) (28.38) 
% enrolled in AP courses only 21.08 31.07 
-9.99 6.14 0.111 
(40.63) (46.28) 
% enrolled in both AP and dual 9.95 5.39 
4.56 2.61 0.087 
enrollment courses (34.72) (22.59) 
% receiving at least one college 36.17 23.56 
, 12.61 3.77 0.002 
credit (47.70) (42.44) 
% receiving at least one college 25.53 12.20 
5 13.33 4.72 0.007 
credit—dual enrollment only (44.43) (32.73) 
% receiving at least one college 7.30 9.97 
: -2.66 2.11 0.213 
credit—AP only (19.71) (29.96) 
a ar 
% receiving at least one college 3.38 1.39 
credit—both AP and dual 2.00 0.85 0.024 
(19.28) (11.70) 
enrollment 
# college credits earned 4.36 2.42 
1.94 0.54 0.001 
(9.33) (6.40) 
# college credits earned—dual 3.42 1.42 
2.01 0.63 0.002 
enrollment only (9.05) (5.05) 
# college credits earned —AP only 0.92 1.01 
-0.08 0.25 0.732 
(2.04) (4.10) 
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